23 May
2021
23 May
'21
5:39 p.m.
Hi guys,
I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will put
it in production on a 1PB+ storage cluster with rgw-only access.
I noticed a weird issue with my mons :
- if I reboot a mon host, the ceph-mon container is not starting after
reboot
- I can see with 'ceph orch ps' the following output :
mon.node01 node01 running (20h) 4m ago
20h 16.2.4 8d91d370c2b8 0a2e86af94b2
mon.node02 node02 running (115m) 12s ago
115m 16.2.4 8d91d370c2b8 51f4885a1b06
mon.node03 node03 stopped 4m ago
19h <unknown> <unknown> <unknown>
(where node03 is the host which was rebooted).
- I tried to start the mon container manually on node03 with '/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run'
and I've got the following output :
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features 3314933069573799936,
adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112,
adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112,
adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112,
adjusting msgr requires
cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164)
36380 : cluster [DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB
data, 605 MiB used, 651 GiB / 652 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s
debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1
mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format 0 -> 3
debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map
reset_timeout 'Monitor::cpu_tp thread 0x7f9a88176700' had timed out
after 0.000000000s
debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0
mon.node03@-1(probing) e5 my rank is now 1 (was -1)
debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing)
e6 removed from monmap, suicide.
root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph
mon.node03 for c2d41ac4-baf5-11eb-865d-2dc838a337a3
Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service;
enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago
Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run
(code=exited, status=0/SUCCESS)
Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited,
status=1/FAILURE)
Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop
(code=exited, status=0/SUCCESS)
Main PID: 1176 (code=exited, status=0/SUCCESS)
The only fix I could find was to redeploy the mon with :
ceph orch daemon rm mon.node03 --force
ceph orch daemon add mon node03
However, even if it's working after redeploy, it's not giving me a lot
of trust to use it in a production environment having an issue like
that. I could reproduce it with 2 different mons so it's not just an
exception.
My setup is based on Ubuntu 20.04 and docker instead of podman :
root@node01:~# docker -v
Docker version 20.10.6, build 370c289
Do you know a workaround for this issue or is this a known bug ? I
noticed that there are some other complaints with the same behaviour in
Octopus as well and the solution at that time was to delete the
/var/lib/ceph/mon folder .
Thanks.
23 May
23 May
11:50 p.m.
Not sure it’s the issue, but it complaina bour msgr not msgr2, do you have the v1 amd v2
adresses in the ceph.conf on that specific osds?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
On 2021. May 23., at 15:40, Adrian Nicolae <adrian.nicolae(a)rcs-rds.ro> wrote:
Hi guys,
I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will put it in
production on a 1PB+ storage cluster with rgw-only access.
I noticed a weird issue with my mons :
- if I reboot a mon host, the ceph-mon container is not starting after reboot
- I can see with 'ceph orch ps' the following output :
mon.node01 node01 running (20h) 4m ago 20h 16.2.4
8d91d370c2b8 0a2e86af94b2
mon.node02 node02 running (115m) 12s ago 115m 16.2.4
8d91d370c2b8 51f4885a1b06
mon.node03 node03 stopped 4m ago 19h
<unknown> <unknown> <unknown>
(where node03 is the host which was rebooted).
- I tried to start the mon container manually on node03 with '/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run' and I've
got the following output :
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 3314933069573799936, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 432629308056666112, adjusting msgr requires
cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164) 36380 : cluster
[DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB data, 605 MiB used, 651 GiB / 652
GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s
debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1 mon.node03(a)-1(???).paxosservice(auth
1..51) refresh upgraded, format 0 -> 3
debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map reset_timeout
'Monitor::cpu_tp thread 0x7f9a88176700' had timed out after 0.000000000s
debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0 mon.node03@-1(probing) e5 my rank is
now 1 (was -1)
debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing) e6 removed from
monmap, suicide.
root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph mon.node03 for
c2d41ac4-baf5-11eb-865d-2dc838a337a3
Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service; enabled; vendor
preset: enabled)
Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago
Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run (code=exited,
status=0/SUCCESS)
Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited, status=1/FAILURE)
Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop (code=exited,
status=0/SUCCESS)
Main PID: 1176 (code=exited, status=0/SUCCESS)
The only fix I could find was to redeploy the mon with :
ceph orch daemon rm mon.node03 --force
ceph orch daemon add mon node03
However, even if it's working after redeploy, it's not giving me a lot of trust to
use it in a production environment having an issue like that. I could reproduce it with 2
different mons so it's not just an exception.
My setup is based on Ubuntu 20.04 and docker instead of podman :
root@node01:~# docker -v
Docker version 20.10.6, build 370c289
Do you know a workaround for this issue or is this a known bug ? I noticed that there are
some other complaints with the same behaviour in Octopus as well and the solution at that
time was to delete the /var/lib/ceph/mon folder .
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
24 May
24 May
1:32 a.m.
It's a fresh Pacific install with the default settings on all hosts :
root@node01:/home/adrian# ceph config show-with-defaults mon.node03 |
grep msgr
mon_warn_on_msgr2_not_enabled true default
ms_bind_msgr1 true default
ms_bind_msgr2 true
On 5/23/2021 5:50 PM, Szabo, Istvan (Agoda) wrote:
Not sure it’s the issue, but it complaina bour msgr
not msgr2, do you
have the v1 amd v2 adresses in the ceph.conf on that specific osds?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com <mailto:istvan.szabo@agoda.com>
---------------------------------------------------
On 2021. May 23., at 15:40, Adrian Nicolae
<adrian.nicolae(a)rcs-rds.ro> wrote:
Hi guys,
I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will
put it in production on a 1PB+ storage cluster with rgw-only access.
I noticed a weird issue with my mons :
- if I reboot a mon host, the ceph-mon container is not starting
after reboot
- I can see with 'ceph orch ps' the following output :
mon.node01 node01 running (20h) 4m
ago 20h 16.2.4 8d91d370c2b8 0a2e86af94b2
mon.node02 node02 running (115m) 12s
ago 115m 16.2.4 8d91d370c2b8 51f4885a1b06
mon.node03 node03 stopped 4m ago 19h
<unknown> <unknown> <unknown>
(where node03 is the host which was rebooted).
- I tried to start the mon container manually on node03 with
'/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run'
and I've got the following output :
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
3314933069573799936, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
432629308056666112, adjusting msgr requires
cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164)
36380 : cluster [DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB
data, 605 MiB used, 651 GiB / 652 GiB avail; 9.6 KiB/s rd, 0 B/s wr,
15 op/s
debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1
mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format
0 -> 3
debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map
reset_timeout 'Monitor::cpu_tp thread 0x7f9a88176700' had timed out
after 0.000000000s
debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0
mon.node03@-1(probing) e5 my rank is now 1 (was -1)
debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0
mon.node03@1(probing) e6 removed from monmap, suicide.
root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph
mon.node03 for c2d41ac4-baf5-11eb-865d-2dc838a337a3
Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service;
enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago
Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run
(code=exited, status=0/SUCCESS)
Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited,
status=1/FAILURE)
Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop
(code=exited, status=0/SUCCESS)
Main PID: 1176 (code=exited, status=0/SUCCESS)
The only fix I could find was to redeploy the mon with :
ceph orch daemon rm mon.node03 --force
ceph orch daemon add mon node03
However, even if it's working after redeploy, it's not giving me a
lot of trust to use it in a production environment having an issue
like that. I could reproduce it with 2 different mons so it's not
just an exception.
My setup is based on Ubuntu 20.04 and docker instead of podman :
root@node01:~# docker -v
Docker version 20.10.6, build 370c289
Do you know a workaround for this issue or is this a known bug ? I
noticed that there are some other complaints with the same behaviour
in Octopus as well and the solution at that time was to delete the
/var/lib/ceph/mon folder .
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
------------------------------------------------------------------------
This message is confidential and is for the sole use of the
intended recipient(s). It may also be privileged or
otherwise protected by copyright or other legal rules. If
you have received it by mistake please let us know by reply
email and delete it from your system. It is prohibited to
copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any
mistaken delivery or unauthorized disclosure of the message.
All messages sent to and from Agoda may be monitored to
ensure compliance with company policies, to protect the
company's interests and to remove potential malware.
Electronic messages may be intercepted, amended, lost or
deleted, or contain viruses.
1:30 a.m.
Hi Adrian,
I have not tried, but I think it will resolve itself automatically after some minutes. How
long have you waited before you do the manual redeploy?
Could you also try “ceph mon dump” to see whether mon.node03 is actually removed from
monmap when it failed to start?
在 2021年5月23日，16:40，Adrian Nicolae
<adrian.nicolae(a)rcs-rds.ro> 写道：
Hi guys,
I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will put it in
production on a 1PB+ storage cluster with rgw-only access.
I noticed a weird issue with my mons :
- if I reboot a mon host, the ceph-mon container is not starting after reboot
- I can see with 'ceph orch ps' the following output :
mon.node01 node01 running (20h) 4m ago 20h 16.2.4
8d91d370c2b8 0a2e86af94b2
mon.node02 node02 running (115m) 12s ago 115m 16.2.4
8d91d370c2b8 51f4885a1b06
mon.node03 node03 stopped 4m ago 19h
<unknown> <unknown> <unknown>
(where node03 is the host which was rebooted).
- I tried to start the mon container manually on node03 with '/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run' and I've
got the following output :
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 3314933069573799936, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map
has features 432629308056666112, adjusting msgr requires
cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164) 36380 : cluster
[DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB data, 605 MiB used, 651 GiB / 652
GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s
debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1 mon.node03(a)-1(???).paxosservice(auth
1..51) refresh upgraded, format 0 -> 3
debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map reset_timeout
'Monitor::cpu_tp thread 0x7f9a88176700' had timed out after 0.000000000s
debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0 mon.node03@-1(probing) e5 my rank is
now 1 (was -1)
debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing) e6 removed from
monmap, suicide.
root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph mon.node03 for
c2d41ac4-baf5-11eb-865d-2dc838a337a3
Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service; enabled; vendor
preset: enabled)
Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago
Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run (code=exited,
status=0/SUCCESS)
Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited, status=1/FAILURE)
Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop (code=exited,
status=0/SUCCESS)
Main PID: 1176 (code=exited, status=0/SUCCESS)
The only fix I could find was to redeploy the mon with :
ceph orch daemon rm mon.node03 --force
ceph orch daemon add mon node03
However, even if it's working after redeploy, it's not giving me a lot of trust
to use it in a production environment having an issue like that. I could reproduce it
with 2 different mons so it's not just an exception.
My setup is based on Ubuntu 20.04 and docker instead of podman :
root@node01:~# docker -v
Docker version 20.10.6, build 370c289
Do you know a workaround for this issue or is this a known bug ? I noticed that there are
some other complaints with the same behaviour in Octopus as well and the solution at that
time was to delete the /var/lib/ceph/mon folder .
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:34 a.m.
Hi,
I waited for more than a day on the first mon failure, it didn't resolve
automatically.
I checked with 'ceph status' and also the ceph.conf on that hosts and
the failed mon was removed from the monmap. The cluster reported only 2
mons (instead of 3) and the third mon was completely removed from config
, it wasn't reported as a failure on 'ceph status'.
On 5/23/2021 7:30 PM, 胡 玮文 wrote:
> Hi Adrian,
>
> I have not tried, but I think it will resolve itself automatically after some
minutes. How long have you waited before you do the manual redeploy?
>
> Could you also try “ceph mon dump” to see whether mon.node03 is actually removed from
monmap when it failed to start?
>
>> 在 2021年5月23日，16:40，Adrian Nicolae <adrian.nicolae(a)rcs-rds.ro> 写道：
>>
>> Hi guys,
>>
>> I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will put it in
production on a 1PB+ storage cluster with rgw-only access.
>>
>> I noticed a weird issue with my mons :
>>
>> - if I reboot a mon host, the ceph-mon container is not starting after reboot
>>
>> - I can see with 'ceph orch ps' the following output :
>>
>> mon.node01 node01 running (20h) 4m ago 20h
16.2.4 8d91d370c2b8 0a2e86af94b2
>> mon.node02 node02 running (115m) 12s ago 115m
16.2.4 8d91d370c2b8 51f4885a1b06
>> mon.node03 node03 stopped 4m ago 19h
<unknown> <unknown> <unknown>
>>
>> (where node03 is the host which was rebooted).
>>
>> - I tried to start the mon container manually on node03 with '/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run' and I've
got the following output :
>>
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 3314933069573799936, adjusting msgr requires
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 432629308056666112, adjusting msgr requires
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 432629308056666112, adjusting msgr requires
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 432629308056666112, adjusting msgr requires
>> cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164) 36380 :
cluster [DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB data, 605 MiB used, 651 GiB
/ 652 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s
>> debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1
mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format 0 -> 3
>> debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map reset_timeout
'Monitor::cpu_tp thread 0x7f9a88176700' had timed out after 0.000000000s
>> debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0 mon.node03@-1(probing) e5 my
rank is now 1 (was -1)
>> debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing) e6
removed from monmap, suicide.
>>
>> root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
>> ● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph mon.node03
for c2d41ac4-baf5-11eb-865d-2dc838a337a3
>> Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service; enabled; vendor
preset: enabled)
>> Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago
>> Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run (code=exited,
status=0/SUCCESS)
>> Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited, status=1/FAILURE)
>> Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop (code=exited,
status=0/SUCCESS)
>> Main PID: 1176 (code=exited, status=0/SUCCESS)
>>
>> The only fix I could find was to redeploy the mon with :
>>
>> ceph orch daemon rm mon.node03 --force
>> ceph orch daemon add mon node03
>>
>> However, even if it's working after redeploy, it's not giving me a lot of
trust to use it in a production environment having an issue like that. I could reproduce
it with 2 different mons so it's not just an exception.
>>
>> My setup is based on Ubuntu 20.04 and docker instead of podman :
>>
>> root@node01:~# docker -v
>> Docker version 20.10.6, build 370c289
>>
>> Do you know a workaround for this issue or is this a known bug ? I noticed that
there are some other complaints with the same behaviour in Octopus as well and the
solution at that time was to delete the /var/lib/ceph/mon folder .
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:52 a.m.
So the orchestrator is aware of that mon is stopped, but not tried to bring it up again.
What is the placement of mon shown in “ceph orch ls”? I explicitly set it to all host
names (e.g. node01;node02;node03), and haven’t experienced this.
在 2021年5月24日，00:35，Adrian Nicolae
<adrian.nicolae(a)rcs-rds.ro> 写道：
Hi,
I waited for more than a day on the first mon failure, it didn't resolve
automatically.
I checked with 'ceph status' and also the ceph.conf on that hosts and the failed
mon was removed from the monmap. The cluster reported only 2 mons (instead of 3) and the
third mon was completely removed from config , it wasn't reported as a failure on
'ceph status'.
> On 5/23/2021 7:30 PM, 胡 玮文 wrote:
> Hi Adrian,
>
> I have not tried, but I think it will resolve itself automatically after some
minutes. How long have you waited before you do the manual redeploy?
>
> Could you also try “ceph mon dump” to see whether mon.node03 is actually removed from
monmap when it failed to start?
>
>>> 在 2021年5月23日，16:40，Adrian Nicolae <adrian.nicolae(a)rcs-rds.ro> 写道：
>>
>> Hi guys,
>>
>> I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will put it in
production on a 1PB+ storage cluster with rgw-only access.
>>
>> I noticed a weird issue with my mons :
>>
>> - if I reboot a mon host, the ceph-mon container is not starting after reboot
>>
>> - I can see with 'ceph orch ps' the following output :
>>
>> mon.node01 node01 running (20h) 4m ago 20h
16.2.4 8d91d370c2b8 0a2e86af94b2
>> mon.node02 node02 running (115m) 12s ago 115m
16.2.4 8d91d370c2b8 51f4885a1b06
>> mon.node03 node03 stopped 4m ago 19h
<unknown> <unknown> <unknown>
>>
>> (where node03 is the host which was rebooted).
>>
>> - I tried to start the mon container manually on node03 with '/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run' and I've
got the following output :
>>
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 3314933069573799936, adjusting msgr requires
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 432629308056666112, adjusting msgr requires
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 432629308056666112, adjusting msgr requires
>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408
crush map has features 432629308056666112, adjusting msgr requires
>> cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164) 36380 :
cluster [DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB data, 605 MiB used, 651 GiB
/ 652 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s
>> debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1
mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format 0 -> 3
>> debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map reset_timeout
'Monitor::cpu_tp thread 0x7f9a88176700' had timed out after 0.000000000s
>> debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0 mon.node03@-1(probing) e5 my
rank is now 1 (was -1)
>> debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing) e6
removed from monmap, suicide.
>>
>> root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
>> ● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph mon.node03
for c2d41ac4-baf5-11eb-865d-2dc838a337a3
>> Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service; enabled; vendor
preset: enabled)
>> Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago
>> Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run (code=exited,
status=0/SUCCESS)
>> Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited, status=1/FAILURE)
>> Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop (code=exited,
status=0/SUCCESS)
>> Main PID: 1176 (code=exited, status=0/SUCCESS)
>>
>> The only fix I could find was to redeploy the mon with :
>>
>> ceph orch daemon rm mon.node03 --force
>> ceph orch daemon add mon node03
>>
>> However, even if it's working after redeploy, it's not giving me a lot of
trust to use it in a production environment having an issue like that. I could reproduce
it with 2 different mons so it's not just an exception.
>>
>> My setup is based on Ubuntu 20.04 and docker instead of podman :
>>
>> root@node01:~# docker -v
>> Docker version 20.10.6, build 370c289
>>
>> Do you know a workaround for this issue or is this a known bug ? I noticed that
there are some other complaints with the same behaviour in Octopus as well and the
solution at that time was to delete the /var/lib/ceph/mon folder .
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:18 a.m.
I think that the orchestrator is trying to bring it up but it's not
starting (see the errors from my previous e-mail) - the container is not
starting even if I tried to start it manually.
the placement is the default one , ceph started the mons automatically
on all my hosts because I only have 3 and the default mon number is 5.
root@node01:/home/adrian# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager 1/1 16m ago 5h count:1
crash 3/3 16m ago 5h *
grafana 1/1 16m ago 5h count:1
mgr 2/2 16m ago 5h count:2
mon 3/5 16m ago 10h count:5
node-exporter 3/3 16m ago 5h *
osd.all-available-devices 12/15 16m ago 4h *
prometheus 1/1 16m ago 5h count:1
rgw.digi1 ?:8000 3/3 16m ago 3h
node01;node02;node03;count:3
I've added the hosts using only the hostnames :
root@node01:/home/adrian# ceph orch host ls
HOST ADDR LABELS STATUS
node01 192.168.80.2
node02 node02
node03 node03
On 5/23/2021 7:52 PM, 胡 玮文 wrote:
> So the orchestrator is aware of that mon is stopped, but not tried to bring it up
again. What is the placement of mon shown in “ceph orch ls”? I explicitly set it to all
host names (e.g. node01;node02;node03), and haven’t experienced this.
>
>> 在 2021年5月24日，00:35，Adrian Nicolae <adrian.nicolae(a)rcs-rds.ro> 写道：
>>
>> Hi,
>>
>> I waited for more than a day on the first mon failure, it didn't resolve
automatically.
>>
>> I checked with 'ceph status' and also the ceph.conf on that hosts and
the failed mon was removed from the monmap. The cluster reported only 2 mons (instead of
3) and the third mon was completely removed from config , it wasn't reported as a
failure on 'ceph status'.
>>
>>
>>> On 5/23/2021 7:30 PM, 胡 玮文 wrote:
>>> Hi Adrian,
>>>
>>> I have not tried, but I think it will resolve itself automatically after some
minutes. How long have you waited before you do the manual redeploy?
>>>
>>> Could you also try “ceph mon dump” to see whether mon.node03 is actually
removed from monmap when it failed to start?
>>>
>>>>> 在 2021年5月23日，16:40，Adrian Nicolae <adrian.nicolae(a)rcs-rds.ro>
写道：
>>>> Hi guys,
>>>>
>>>> I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will
put it in production on a 1PB+ storage cluster with rgw-only access.
>>>>
>>>> I noticed a weird issue with my mons :
>>>>
>>>> - if I reboot a mon host, the ceph-mon container is not starting after
reboot
>>>>
>>>> - I can see with 'ceph orch ps' the following output :
>>>>
>>>> mon.node01 node01 running (20h) 4m ago
20h 16.2.4 8d91d370c2b8 0a2e86af94b2
>>>> mon.node02 node02 running (115m) 12s ago
115m 16.2.4 8d91d370c2b8 51f4885a1b06
>>>> mon.node03 node03 stopped 4m ago
19h <unknown> <unknown> <unknown>
>>>>
>>>> (where node03 is the host which was rebooted).
>>>>
>>>> - I tried to start the mon container manually on node03 with
'/bin/bash /var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run'
and I've got the following output :
>>>>
>>>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd
e408 crush map has features 3314933069573799936, adjusting msgr requires
>>>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd
e408 crush map has features 432629308056666112, adjusting msgr requires
>>>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd
e408 crush map has features 432629308056666112, adjusting msgr requires
>>>> debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd
e408 crush map has features 432629308056666112, adjusting msgr requires
>>>> cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164)
36380 : cluster [DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB data, 605 MiB used,
651 GiB / 652 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s
>>>> debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1
mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format 0 -> 3
>>>> debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map
reset_timeout 'Monitor::cpu_tp thread 0x7f9a88176700' had timed out after
0.000000000s
>>>> debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0 mon.node03@-1(probing)
e5 my rank is now 1 (was -1)
>>>> debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing)
e6 removed from monmap, suicide.
>>>>
>>>> root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
>>>> ● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph
mon.node03 for c2d41ac4-baf5-11eb-865d-2dc838a337a3
>>>> Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service; enabled; vendor
preset: enabled)
>>>> Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min
ago
>>>> Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run (code=exited,
status=0/SUCCESS)
>>>> Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited, status=1/FAILURE)
>>>> Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop (code=exited,
status=0/SUCCESS)
>>>> Main PID: 1176 (code=exited, status=0/SUCCESS)
>>>>
>>>> The only fix I could find was to redeploy the mon with :
>>>>
>>>> ceph orch daemon rm mon.node03 --force
>>>> ceph orch daemon add mon node03
>>>>
>>>> However, even if it's working after redeploy, it's not giving me
a lot of trust to use it in a production environment having an issue like that. I could
reproduce it with 2 different mons so it's not just an exception.
>>>>
>>>> My setup is based on Ubuntu 20.04 and docker instead of podman :
>>>>
>>>> root@node01:~# docker -v
>>>> Docker version 20.10.6, build 370c289
>>>>
>>>> Do you know a workaround for this issue or is this a known bug ? I
noticed that there are some other complaints with the same behaviour in Octopus as well
and the solution at that time was to delete the /var/lib/ceph/mon folder .
>>>>
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 May
25 May
11:15 p.m.
Hi,
I wanted to explore the stretch mode in pacific (16.2.4) and see how
it behaves with a DC failure. It seems as if I'm hitting the same or
at least a similar issue here. To verify if it's the stretch mode I
removed the cluster and rebuilt it without stretch mode, three hosts
in three DCs and started to reboot. First I rebooted one node, the
cluster came back to HEALTH_OK. Then I rebooted two of the three nodes
and again everything recovered successfully.
Then I rebuilt a 5 node cluster, two DCs in stretch mode with three
MONs, one being a tiebreaker in a virtual third DC. The stretch rule
was applied (4 replicas across all 4 nodes).
To test a DC failure I simply shut down two nodes from DC2, although
the pool's min_size was reduced to 1 by ceph I couldn't read or write
anything to a mapped rbd, althouh ceph still was responsive with two
active MONs.
When I booted the other two nodes again the cluster was not able to
recover, it ends up in a loop of restarting the MON containers (the
OSDs recover eventually) until systemd shuts them down due to too many
restarts.
For a couple of seconds I get a ceph status, but I never get all three
MONs up. When there are two MONs up and I restart the missing one a
different MON is shut down.
I also see the error message mentioned here in this thread
heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7ff3b3aa5700'
had timed out after 0.000000000s
I'll add some more information, a stack trace from MON failure:
---snip---
2021-05-25T15:44:26.988562+02:00 pacific1 conmon[5132]: 5
mon.pacific1(a)0(leader).paxos(paxos updating c 9288..9839) is_readable
= 1 - now=2021-05-25T13:44:26.730359+0000
lease_expire=2021-05-25T13:44:30.270907+0000 has v0 lc 9839
2021-05-25T15:44:26.988638+02:00 pacific1 conmon[5132]: debug -5>
2021-05-25T13:44:26.726+0000 7ff3b1aa1700 2 mon.pacific1@0(leader)
e13 send_reply 0x55e37aae3860 0x55e37affa9c0 auth_reply(proto 2 0 (0)
Success) v1
2021-05-25T15:44:26.988714+02:00 pacific1 conmon[5132]: debug -4>
2021-05-25T13:44:26.726+0000 7ff3b1aa1700 5
mon.pacific1(a)0(leader).paxos(paxos updating c 9288..9839) is_readable
= 1 - now=2021-05-25T13:44:26.731084+0000
lease_expire=2021-05-25T13:44:30.270907+0000 has v0 lc 9839
2021-05-25T15:44:26.988790+02:00 pacific1 conmon[5132]: debug -3>
2021-05-25T13:44:26.726+0000 7ff3b1aa1700 2 mon.pacific1@0(leader)
e13 send_reply 0x55e37b14def0 0x55e37ab11ba0 auth_reply(proto 2 0 (0)
Success) v1
2021-05-25T15:44:26.988929+02:00 pacific1 conmon[5132]: debug -2>
2021-05-25T13:44:26.730+0000 7ff3b1aa1700 5
mon.pacific1(a)0(leader).osd e117 send_incremental [105..117] to
client.84146
2021-05-25T15:44:26.989012+02:00 pacific1 conmon[5132]: debug -1>
2021-05-25T13:44:26.734+0000 7ff3b1aa1700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/osd/OSDMap.cc:
In function 'void OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&,
uint64_t) const' thread 7ff3b1aa1700 time
2021-05-25T13:44:26.732857+0000
2021-05-25T15:44:26.989087+02:00 pacific1 conmon[5132]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/osd/OSDMap.cc:
658: FAILED ceph_assert(target_v >=
9)
2021-05-25T15:44:26.989163+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.989239+02:00 pacific1 conmon[5132]: ceph version
16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
2021-05-25T15:44:26.989314+02:00 pacific1 conmon[5132]: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x158) [0x7ff3bf61a59c]
2021-05-25T15:44:26.989388+02:00 pacific1 conmon[5132]: 2:
/usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7ff3bf61a7b6]
2021-05-25T15:44:26.989489+02:00 pacific1 conmon[5132]: 3:
(OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&, unsigned
long) const+0x539) [0x7ff3bfa529f9]
2021-05-25T15:44:26.989560+02:00 pacific1 conmon[5132]: 4:
(OSDMonitor::reencode_incremental_map(ceph::buffer::v15_2_0::list&,
unsigned long)+0x1c9) [0x55e377b36df9]
2021-05-25T15:44:26.989627+02:00 pacific1 conmon[5132]: 5:
(OSDMonitor::get_version(unsigned long, unsigned long,
ceph::buffer::v15_2_0::list&)+0x1f4) [0x55e377b37234]
2021-05-25T15:44:26.989693+02:00 pacific1 conmon[5132]: 6:
(OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned
long)+0x301) [0x55e377b3a3c1]
2021-05-25T15:44:26.989759+02:00 pacific1 conmon[5132]: 7:
(OSDMonitor::send_incremental(unsigned int, MonSession*, bool,
boost::intrusive_ptr<MonOpRequest>)+0x104) [0x55e377b3b094]
2021-05-25T15:44:26.989825+02:00 pacific1 conmon[5132]: 8:
(OSDMonitor::check_osdmap_sub(Subscription*)+0x72) [0x55e377b42792]
2021-05-25T15:44:26.989891+02:00 pacific1 conmon[5132]: 9:
(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xe82)
[0x55e3779da402]
2021-05-25T15:44:26.989967+02:00 pacific1 conmon[5132]: 10:
(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d)
[0x55e377a002ed]
2021-05-25T15:44:26.990046+02:00 pacific1 conmon[5132]: 11:
(Monitor::_ms_dispatch(Message*)+0x670) [0x55e377a01910]
2021-05-25T15:44:26.990113+02:00 pacific1 conmon[5132]: 12:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c)
[0x55e377a2ffdc]
2021-05-25T15:44:26.990179+02:00 pacific1 conmon[5132]: 13:
(DispatchQueue::entry()+0x126a) [0x7ff3bf854b1a]
2021-05-25T15:44:26.990255+02:00 pacific1 conmon[5132]: 14:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3bf904b71]
2021-05-25T15:44:26.990330+02:00 pacific1 conmon[5132]: 15:
/lib64/libpthread.so.0(+0x814a) [0x7ff3bd10a14a]
2021-05-25T15:44:26.990420+02:00 pacific1 conmon[5132]: 16: clone()
2021-05-25T15:44:26.990497+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.990573+02:00 pacific1 conmon[5132]: debug 0>
2021-05-25T13:44:26.742+0000 7ff3b1aa1700 -1 *** Caught signal
(Aborted) **
2021-05-25T15:44:26.990648+02:00 pacific1 conmon[5132]: in thread
7ff3b1aa1700 thread_name:ms_dispatch
2021-05-25T15:44:26.990723+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.990806+02:00 pacific1 conmon[5132]: ceph version
16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
2021-05-25T15:44:26.990883+02:00 pacific1 conmon[5132]: 1:
/lib64/libpthread.so.0(+0x12b20) [0x7ff3bd114b20]
2021-05-25T15:44:26.990958+02:00 pacific1 conmon[5132]: 2: gsignal()
2021-05-25T15:44:26.991034+02:00 pacific1 conmon[5132]: 3: abort()
2021-05-25T15:44:26.991110+02:00 pacific1 conmon[5132]: 4:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1a9) [0x7ff3bf61a5ed]
2021-05-25T15:44:26.991176+02:00 pacific1 conmon[5132]: 5:
/usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7ff3bf61a7b6]
2021-05-25T15:44:26.991251+02:00 pacific1 conmon[5132]: 6:
(OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&, unsigned
long) const+0x539) [0x7ff3bfa529f9]
2021-05-25T15:44:26.991326+02:00 pacific1 conmon[5132]: 7:
(OSDMonitor::reencode_incremental_map(ceph::buffer::v15_2_0::list&,
unsigned long)+0x1c9) [0x55e377b36df9]
2021-05-25T15:44:26.991393+02:00 pacific1 conmon[5132]: 8:
(OSDMonitor::get_version(unsigned long, unsigned long,
ceph::buffer::v15_2_0::list&)+0x1f4) [0x55e377b37234]
2021-05-25T15:44:26.991460+02:00 pacific1 conmon[5132]: 9:
(OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned
long)+0x301) [0x55e377b3a3c1]
2021-05-25T15:44:26.991557+02:00 pacific1 conmon[5132]: 10:
(OSDMonitor::send_incremental(unsigned int, MonSession*, bool,
boost::intrusive_ptr<MonOpRequest>)+0x104) [0x55e377b3b094]
2021-05-25T15:44:26.991628+02:00 pacific1 conmon[5132]: 11:
(OSDMonitor::check_osdmap_sub(Subscription*)+0x72) [0x55e377b42792]
2021-05-25T15:44:26.991695+02:00 pacific1 conmon[5132]: 12:
(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xe82)
[0x55e3779da402]
2021-05-25T15:44:26.991761+02:00 pacific1 conmon[5132]: 13:
(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d)
[0x55e377a002ed]
2021-05-25T15:44:26.991827+02:00 pacific1 conmon[5132]: 14:
(Monitor::_ms_dispatch(Message*)+0x670) [0x55e377a01910]
2021-05-25T15:44:26.991893+02:00 pacific1 conmon[5132]: 15:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c)
[0x55e377a2ffdc]
2021-05-25T15:44:26.991959+02:00 pacific1 conmon[5132]: 16:
(DispatchQueue::entry()+0x126a) [0x7ff3bf854b1a]
2021-05-25T15:44:26.992025+02:00 pacific1 conmon[5132]: 17:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3bf904b71]
2021-05-25T15:44:26.992091+02:00 pacific1 conmon[5132]: 18:
/lib64/libpthread.so.0(+0x814a) [0x7ff3bd10a14a]
2021-05-25T15:44:26.992156+02:00 pacific1 conmon[5132]: 19: clone()
---snip---
I can't tell if this is due to the limited resources in my virtual
cluster but I figured since the non-stretch mode seems to work as
expected this could be a problem with the stretch mode. I can provide
more information if required, just let me know what I can do.
Regards,
Eugen
[1] https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
Zitat von Adrian Nicolae <adrian.nicolae(a)rcs-rds.ro>ro>:
Hi guys,
I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will
put it in production on a 1PB+ storage cluster with rgw-only access.
I noticed a weird issue with my mons :
- if I reboot a mon host, the ceph-mon container is not starting after reboot
- I can see with 'ceph orch ps' the following output :
mon.node01 node01 running (20h) 4m
ago 20h 16.2.4 8d91d370c2b8 0a2e86af94b2
mon.node02 node02 running (115m) 12s
ago 115m 16.2.4 8d91d370c2b8 51f4885a1b06
mon.node03 node03 stopped 4m
ago 19h <unknown> <unknown> <unknown>
(where node03 is the host which was rebooted).
- I tried to start the mon container manually on node03 with
'/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run' and
I've got the following output :
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
3314933069573799936, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
432629308056666112, adjusting msgr requires
debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0
mon.node03(a)-1(???).osd e408 crush map has features
432629308056666112, adjusting msgr requires
cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls
(mgr.14164) 36380 : cluster [DBG] pgmap v36392: 417 pgs: 417
active+clean; 33 KiB data, 605 MiB used, 651 GiB / 652 GiB avail;
9.6 KiB/s rd, 0 B/s wr, 15 op/s
debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1
mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format
0 -> 3
debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map
reset_timeout 'Monitor::cpu_tp thread 0x7f9a88176700' had timed out
after 0.000000000s
debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0
mon.node03@-1(probing) e5 my rank is now 1 (was -1)
debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0
mon.node03@1(probing) e6 removed from monmap, suicide.
root@node03:/home/adrian# systemctl status
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service
● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service -
Ceph mon.node03 for c2d41ac4-baf5-11eb-865d-2dc838a337a3
Loaded: loaded
(/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service;
enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago
Process: 1176 ExecStart=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run
(code=exited, status=0/SUCCESS)
Process: 1855 ExecStop=/usr/bin/docker stop
ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited,
status=1/FAILURE)
Process: 1861 ExecStopPost=/bin/bash
/var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop
(code=exited, status=0/SUCCESS)
Main PID: 1176 (code=exited, status=0/SUCCESS)
The only fix I could find was to redeploy the mon with :
ceph orch daemon rm mon.node03 --force
ceph orch daemon add mon node03
However, even if it's working after redeploy, it's not giving me a
lot of trust to use it in a production environment having an issue
like that. I could reproduce it with 2 different mons so it's not
just an exception.
My setup is based on Ubuntu 20.04 and docker instead of podman :
root@node01:~# docker -v
Docker version 20.10.6, build 370c289
Do you know a workaround for this issue or is this a known bug ? I
noticed that there are some other complaints with the same behaviour
in Octopus as well and the solution at that time was to delete the
/var/lib/ceph/mon folder .
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 May
26 May
12:56 a.m.
On Tue, May 25, 2021 at 7:17 AM Eugen Block <eblock(a)nde.ag> wrote:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/osd/OSDMap.cc:
In function 'void OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&,
uint64_t) const' thread 7ff3b1aa1700 time
2021-05-25T13:44:26.732857+0000
2021-05-25T15:44:26.989087+02:00 pacific1 conmon[5132]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/osd/OSDMap.cc:
658: FAILED ceph_assert(target_v >=
9)
2021-05-25T15:44:26.989163+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.989239+02:00 pacific1 conmon[5132]: ceph version
16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
2021-05-25T15:44:26.989314+02:00 pacific1 conmon[5132]: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x158) [0x7ff3bf61a59c]
2021-05-25T15:44:26.989388+02:00 pacific1 conmon[5132]: 2:
/usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7ff3bf61a7b6]
2021-05-25T15:44:26.989489+02:00 pacific1 conmon[5132]: 3:
(OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&, unsigned
long) const+0x539) [0x7ff3bfa529f9]
2021-05-25T15:44:26.989560+02:00 pacific1 conmon[5132]: 4:
(OSDMonitor::reencode_incremental_map(ceph::buffer::v15_2_0::list&,
unsigned long)+0x1c9) [0x55e377b36df9]
2021-05-25T15:44:26.989627+02:00 pacific1 conmon[5132]: 5:
(OSDMonitor::get_version(unsigned long, unsigned long,
ceph::buffer::v15_2_0::list&)+0x1f4) [0x55e377b37234]
2021-05-25T15:44:26.989693+02:00 pacific1 conmon[5132]: 6:
(OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned
long)+0x301) [0x55e377b3a3c1]
2021-05-25T15:44:26.989759+02:00 pacific1 conmon[5132]: 7:
(OSDMonitor::send_incremental(unsigned int, MonSession*, bool,
boost::intrusive_ptr<MonOpRequest>)+0x104) [0x55e377b3b094]
2021-05-25T15:44:26.989825+02:00 pacific1 conmon[5132]: 8:
(OSDMonitor::check_osdmap_sub(Subscription*)+0x72) [0x55e377b42792]
2021-05-25T15:44:26.989891+02:00 pacific1 conmon[5132]: 9:
(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xe82)
[0x55e3779da402]
2021-05-25T15:44:26.989967+02:00 pacific1 conmon[5132]: 10:
(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d)
[0x55e377a002ed]
2021-05-25T15:44:26.990046+02:00 pacific1 conmon[5132]: 11:
(Monitor::_ms_dispatch(Message*)+0x670) [0x55e377a01910]
2021-05-25T15:44:26.990113+02:00 pacific1 conmon[5132]: 12:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c)
[0x55e377a2ffdc]
2021-05-25T15:44:26.990179+02:00 pacific1 conmon[5132]: 13:
(DispatchQueue::entry()+0x126a) [0x7ff3bf854b1a]
2021-05-25T15:44:26.990255+02:00 pacific1 conmon[5132]: 14:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3bf904b71]
2021-05-25T15:44:26.990330+02:00 pacific1 conmon[5132]: 15:
/lib64/libpthread.so.0(+0x814a) [0x7ff3bd10a14a]
2021-05-25T15:44:26.990420+02:00 pacific1 conmon[5132]: 16: clone()
2021-05-25T15:44:26.990497+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.990573+02:00 pacific1 conmon[5132]: debug 0>
2021-05-25T13:44:26.742+0000 7ff3b1aa1700 -1 *** Caught signal
(Aborted) **
2021-05-25T15:44:26.990648+02:00 pacific1 conmon[5132]: in thread
7ff3b1aa1700 thread_name:ms_dispatch
2021-05-25T15:44:26.990723+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.990806+02:00 pacific1 conmon[5132]: ceph version
16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
2021-05-25T15:44:26.990883+02:00 pacific1 conmon[5132]: 1:
/lib64/libpthread.so.0(+0x12b20) [0x7ff3bd114b20]
2021-05-25T15:44:26.990958+02:00 pacific1 conmon[5132]: 2: gsignal()
2021-05-25T15:44:26.991034+02:00 pacific1 conmon[5132]: 3: abort()
2021-05-25T15:44:26.991110+02:00 pacific1 conmon[5132]: 4:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1a9) [0x7ff3bf61a5ed]
2021-05-25T15:44:26.991176+02:00 pacific1 conmon[5132]: 5:
/usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7ff3bf61a7b6]
2021-05-25T15:44:26.991251+02:00 pacific1 conmon[5132]: 6:
(OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&, unsigned
long) const+0x539) [0x7ff3bfa529f9]
2021-05-25T15:44:26.991326+02:00 pacific1 conmon[5132]: 7:
(OSDMonitor::reencode_incremental_map(ceph::buffer::v15_2_0::list&,
unsigned long)+0x1c9) [0x55e377b36df9]
2021-05-25T15:44:26.991393+02:00 pacific1 conmon[5132]: 8:
(OSDMonitor::get_version(unsigned long, unsigned long,
ceph::buffer::v15_2_0::list&)+0x1f4) [0x55e377b37234]
2021-05-25T15:44:26.991460+02:00 pacific1 conmon[5132]: 9:
(OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned
long)+0x301) [0x55e377b3a3c1]
2021-05-25T15:44:26.991557+02:00 pacific1 conmon[5132]: 10:
(OSDMonitor::send_incremental(unsigned int, MonSession*, bool,
boost::intrusive_ptr<MonOpRequest>)+0x104) [0x55e377b3b094]
2021-05-25T15:44:26.991628+02:00 pacific1 conmon[5132]: 11:
(OSDMonitor::check_osdmap_sub(Subscription*)+0x72) [0x55e377b42792]
2021-05-25T15:44:26.991695+02:00 pacific1 conmon[5132]: 12:
(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xe82)
[0x55e3779da402]
2021-05-25T15:44:26.991761+02:00 pacific1 conmon[5132]: 13:
(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d)
[0x55e377a002ed]
2021-05-25T15:44:26.991827+02:00 pacific1 conmon[5132]: 14:
(Monitor::_ms_dispatch(Message*)+0x670) [0x55e377a01910]
2021-05-25T15:44:26.991893+02:00 pacific1 conmon[5132]: 15:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c)
[0x55e377a2ffdc]
2021-05-25T15:44:26.991959+02:00 pacific1 conmon[5132]: 16:
(DispatchQueue::entry()+0x126a) [0x7ff3bf854b1a]
2021-05-25T15:44:26.992025+02:00 pacific1 conmon[5132]: 17:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3bf904b71]
2021-05-25T15:44:26.992091+02:00 pacific1 conmon[5132]: 18:
/lib64/libpthread.so.0(+0x814a) [0x7ff3bd10a14a]
2021-05-25T15:44:26.992156+02:00 pacific1 conmon[5132]: 19: clone()
---snip---
I can't tell if this is due to the limited resources in my virtual
cluster but I figured since the non-stretch mode seems to work as
expected this could be a problem with the stretch mode. I can provide
more information if required, just let me know what I can do.
This crash is an issue with overly-zealous safety checks breaking when
encoding the osdmap for kernel clients in stretch mode, and is fixed
for the next pacific point release[1]. Sorry for the trouble.
-Greg
[1]:
https://github.com/ceph/ceph/pull/40484/commits/9453e34ea7c480ddfea8363cbad…
1:06 a.m.
Thanks for the confirmation, Greg! I‘ll try with a newer release then.
That’s why we’re testing, isn’t it? ;-)
Then the OPs issue is probably not resolved yet since he didn’t
mention a stretch cluster. Sorry for high-jacking the thread.
Zitat von Gregory Farnum <gfarnum(a)redhat.com>om>:
On Tue, May 25, 2021 at 7:17 AM Eugen Block
<eblock(a)nde.ag> wrote:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/osd/OSDMap.cc:
In function 'void OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&,
uint64_t) const' thread 7ff3b1aa1700
time
2021-05-25T13:44:26.732857+0000
2021-05-25T15:44:26.989087+02:00 pacific1 conmon[5132]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/osd/OSDMap.cc:
658: FAILED ceph_assert(target_v
This crash is an issue with overly-zealous safety checks breaking when
encoding the osdmap for kernel clients in stretch mode, and is fixed
for the next pacific point release[1]. Sorry for the trouble.
-Greg
[1]:
https://github.com/ceph/ceph/pull/40484/commits/9453e34ea7c480ddfea8363cbad…
=
9)
2021-05-25T15:44:26.989163+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.989239+02:00 pacific1 conmon[5132]: ceph version
16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
2021-05-25T15:44:26.989314+02:00 pacific1 conmon[5132]: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x158) [0x7ff3bf61a59c]
2021-05-25T15:44:26.989388+02:00 pacific1 conmon[5132]: 2:
/usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7ff3bf61a7b6]
2021-05-25T15:44:26.989489+02:00 pacific1 conmon[5132]: 3:
(OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&, unsigned
long) const+0x539) [0x7ff3bfa529f9]
2021-05-25T15:44:26.989560+02:00 pacific1 conmon[5132]: 4:
(OSDMonitor::reencode_incremental_map(ceph::buffer::v15_2_0::list&,
unsigned long)+0x1c9) [0x55e377b36df9]
2021-05-25T15:44:26.989627+02:00 pacific1 conmon[5132]: 5:
(OSDMonitor::get_version(unsigned long, unsigned long,
ceph::buffer::v15_2_0::list&)+0x1f4) [0x55e377b37234]
2021-05-25T15:44:26.989693+02:00 pacific1 conmon[5132]: 6:
(OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned
long)+0x301) [0x55e377b3a3c1]
2021-05-25T15:44:26.989759+02:00 pacific1 conmon[5132]: 7:
(OSDMonitor::send_incremental(unsigned int, MonSession*, bool,
boost::intrusive_ptr<MonOpRequest>)+0x104) [0x55e377b3b094]
2021-05-25T15:44:26.989825+02:00 pacific1 conmon[5132]: 8:
(OSDMonitor::check_osdmap_sub(Subscription*)+0x72) [0x55e377b42792]
2021-05-25T15:44:26.989891+02:00 pacific1 conmon[5132]: 9:
(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xe82)
[0x55e3779da402]
2021-05-25T15:44:26.989967+02:00 pacific1 conmon[5132]: 10:
(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d)
[0x55e377a002ed]
2021-05-25T15:44:26.990046+02:00 pacific1 conmon[5132]: 11:
(Monitor::_ms_dispatch(Message*)+0x670) [0x55e377a01910]
2021-05-25T15:44:26.990113+02:00 pacific1 conmon[5132]: 12:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c)
[0x55e377a2ffdc]
2021-05-25T15:44:26.990179+02:00 pacific1 conmon[5132]: 13:
(DispatchQueue::entry()+0x126a) [0x7ff3bf854b1a]
2021-05-25T15:44:26.990255+02:00 pacific1 conmon[5132]: 14:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3bf904b71]
2021-05-25T15:44:26.990330+02:00 pacific1 conmon[5132]: 15:
/lib64/libpthread.so.0(+0x814a) [0x7ff3bd10a14a]
2021-05-25T15:44:26.990420+02:00 pacific1 conmon[5132]: 16: clone()
2021-05-25T15:44:26.990497+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.990573+02:00 pacific1 conmon[5132]: debug 0>
2021-05-25T13:44:26.742+0000 7ff3b1aa1700 -1 *** Caught signal
(Aborted) **
2021-05-25T15:44:26.990648+02:00 pacific1 conmon[5132]: in thread
7ff3b1aa1700 thread_name:ms_dispatch
2021-05-25T15:44:26.990723+02:00 pacific1 conmon[5132]:
2021-05-25T15:44:26.990806+02:00 pacific1 conmon[5132]: ceph version
16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
2021-05-25T15:44:26.990883+02:00 pacific1 conmon[5132]: 1:
/lib64/libpthread.so.0(+0x12b20) [0x7ff3bd114b20]
2021-05-25T15:44:26.990958+02:00 pacific1 conmon[5132]: 2: gsignal()
2021-05-25T15:44:26.991034+02:00 pacific1 conmon[5132]: 3: abort()
2021-05-25T15:44:26.991110+02:00 pacific1 conmon[5132]: 4:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1a9) [0x7ff3bf61a5ed]
2021-05-25T15:44:26.991176+02:00 pacific1 conmon[5132]: 5:
/usr/lib64/ceph/libceph-common.so.2(+0x2767b6) [0x7ff3bf61a7b6]
2021-05-25T15:44:26.991251+02:00 pacific1 conmon[5132]: 6:
(OSDMap::Incremental::encode(ceph::buffer::v15_2_0::list&, unsigned
long) const+0x539) [0x7ff3bfa529f9]
2021-05-25T15:44:26.991326+02:00 pacific1 conmon[5132]: 7:
(OSDMonitor::reencode_incremental_map(ceph::buffer::v15_2_0::list&,
unsigned long)+0x1c9) [0x55e377b36df9]
2021-05-25T15:44:26.991393+02:00 pacific1 conmon[5132]: 8:
(OSDMonitor::get_version(unsigned long, unsigned long,
ceph::buffer::v15_2_0::list&)+0x1f4) [0x55e377b37234]
2021-05-25T15:44:26.991460+02:00 pacific1 conmon[5132]: 9:
(OSDMonitor::build_incremental(unsigned int, unsigned int, unsigned
long)+0x301) [0x55e377b3a3c1]
2021-05-25T15:44:26.991557+02:00 pacific1 conmon[5132]: 10:
(OSDMonitor::send_incremental(unsigned int, MonSession*, bool,
boost::intrusive_ptr<MonOpRequest>)+0x104) [0x55e377b3b094]
2021-05-25T15:44:26.991628+02:00 pacific1 conmon[5132]: 11:
(OSDMonitor::check_osdmap_sub(Subscription*)+0x72) [0x55e377b42792]
2021-05-25T15:44:26.991695+02:00 pacific1 conmon[5132]: 12:
(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xe82)
[0x55e3779da402]
2021-05-25T15:44:26.991761+02:00 pacific1 conmon[5132]: 13:
(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x78d)
[0x55e377a002ed]
2021-05-25T15:44:26.991827+02:00 pacific1 conmon[5132]: 14:
(Monitor::_ms_dispatch(Message*)+0x670) [0x55e377a01910]
2021-05-25T15:44:26.991893+02:00 pacific1 conmon[5132]: 15:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c)
[0x55e377a2ffdc]
2021-05-25T15:44:26.991959+02:00 pacific1 conmon[5132]: 16:
(DispatchQueue::entry()+0x126a) [0x7ff3bf854b1a]
2021-05-25T15:44:26.992025+02:00 pacific1 conmon[5132]: 17:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3bf904b71]
2021-05-25T15:44:26.992091+02:00 pacific1 conmon[5132]: 18:
/lib64/libpthread.so.0(+0x814a) [0x7ff3bd10a14a]
2021-05-25T15:44:26.992156+02:00 pacific1 conmon[5132]: 19: clone()
---snip---
I can't tell if this is due to the limited resources in my virtual
cluster but I figured since the non-stretch mode seems to work as
expected this could be a problem with the stretch mode. I can provide
more information if required, just let me know what I can do.
1:42 a.m.
Hi,
On my setup I didn't enable a strech cluster. It's just a 3 x VM setup
running on the same Proxmox node, all the nodes are using a single
unique network. I installed Ceph using the documented cephadm flow.
Thanks for the confirmation, Greg! I‘ll try with a
newer release then. >That’s why we’re testing, isn’t it? ;-) >Then the OPs issue
is
probably not resolved yet since he didn’t >mention a stretch cluster.
Sorry for high-jacking the thread.
1070
days inactive
1072
days old
10 comments
5 participants
participants (5)
-
Adrian Nicolae -
Eugen Block -
Gregory Farnum -
Szabo, Istvan (Agoda) -
胡 玮文