[RGW] radosgw does not respond after some time after upgrade from pacific to quincy
Hi, we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything seems to work, however after some time radosgw stops responding and we have to restart it. At first look, it seems that radosgw stops responding sometimes during recovery. Does this maybe have to do something with mclock https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ? ceph.conf looks something like this: -------- [global] fsid = ... mon initial members = x1,x2 mon host = # public network = 192.168.0.0/24 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd crush chooseleaf type = 1 [osd] osd_scrub_begin_hour = 18 osd_scrub_end_hour = 6 osd_class_update_on_start = false osd_scrub_during_recovery = false #scrub during recovery osd_scrub_max_interval = 1209600 osd_deep_scrub_interval = 1209600 osd_max_scrubs = 3 osd_scrub_load_threshold = 1 [client.radosgw.mon2] host = mon2 # keyring = /etc/ceph/ceph.client.radosgw.keyring log_file = /var/log/ceph/client.radosgw.mon2.log rgw_dns_name = ... rgw_frontends = "beast port=4444" rgw_max_put_param_size = 15728640 rgw_crypt_require_ssl = false rgw_max_concurrent_requests = 2048 -------- We have nginx in front of rgw which upstream is set to client.radosgw.mon2 port 4444. Kind regards, Rok
What I only see in rgw logs in that "hang time" is something like: 2024-07-23T20:00:45.666+0200 7fc751496700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-07-23T20:00:57.072+0200 7fc740c75700 20 rgw notify: INFO: next queues processing will happen at: Tue Jul 23 20:01:27 2024 2024-07-23T20:01:07.666+0200 7fc751496700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-07-23T20:01:27.534+0200 7fc740c75700 20 rgw notify: INFO: next queues processing will happen at: Tue Jul 23 20:01:57 2024 On Tue, Jul 23, 2024 at 7:58 PM Rok Jaklič <rjaklic@gmail.com> wrote:
Hi,
we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything seems to work, however after some time radosgw stops responding and we have to restart it.
At first look, it seems that radosgw stops responding sometimes during recovery.
Does this maybe have to do something with mclock https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ?
ceph.conf looks something like this:
--------
[global] fsid = ... mon initial members = x1,x2 mon host = # public network = 192.168.0.0/24 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc
osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd crush chooseleaf type = 1
[osd] osd_scrub_begin_hour = 18 osd_scrub_end_hour = 6 osd_class_update_on_start = false osd_scrub_during_recovery = false #scrub during recovery osd_scrub_max_interval = 1209600 osd_deep_scrub_interval = 1209600 osd_max_scrubs = 3 osd_scrub_load_threshold = 1
[client.radosgw.mon2] host = mon2 # keyring = /etc/ceph/ceph.client.radosgw.keyring log_file = /var/log/ceph/client.radosgw.mon2.log rgw_dns_name = ... rgw_frontends = "beast port=4444" rgw_max_put_param_size = 15728640 rgw_crypt_require_ssl = false rgw_max_concurrent_requests = 2048 --------
We have nginx in front of rgw which upstream is set to client.radosgw.mon2 port 4444.
Kind regards, Rok
Hi, can you tell a bit more about your setup? Are RGWs and OSDs colocated on the same servers? Are there any signs of a server overload like OOM killers or anything else related to the recovery? Are disks saturated? Is this cephadm managed? What's the current ceph status? Thanks, Eugen Zitat von Rok Jaklič <rjaklic@gmail.com>:
Hi,
we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything seems to work, however after some time radosgw stops responding and we have to restart it.
At first look, it seems that radosgw stops responding sometimes during recovery.
Does this maybe have to do something with mclock https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ?
ceph.conf looks something like this:
--------
[global] fsid = ... mon initial members = x1,x2 mon host = # public network = 192.168.0.0/24 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc
osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd crush chooseleaf type = 1
[osd] osd_scrub_begin_hour = 18 osd_scrub_end_hour = 6 osd_class_update_on_start = false osd_scrub_during_recovery = false #scrub during recovery osd_scrub_max_interval = 1209600 osd_deep_scrub_interval = 1209600 osd_max_scrubs = 3 osd_scrub_load_threshold = 1
[client.radosgw.mon2] host = mon2 # keyring = /etc/ceph/ceph.client.radosgw.keyring log_file = /var/log/ceph/client.radosgw.mon2.log rgw_dns_name = ... rgw_frontends = "beast port=4444" rgw_max_put_param_size = 15728640 rgw_crypt_require_ssl = false rgw_max_concurrent_requests = 2048 --------
We have nginx in front of rgw which upstream is set to client.radosgw.mon2 port 4444.
Kind regards, Rok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It turned out that under heavy load one of the rgws have had too low soft NOFILE limit set (max number of open files). We also accidentally found that in our setup and hw the "round trips" under heavy load from frontend proxy (nginx in our case) to radosgw cost is pretty high (let us say 50% in capability of sent p/s), even if it is on the same machine and ip. Thanks anyway. On Wed, Jul 24, 2024 at 8:25 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
can you tell a bit more about your setup? Are RGWs and OSDs colocated on the same servers? Are there any signs of a server overload like OOM killers or anything else related to the recovery? Are disks saturated? Is this cephadm managed? What's the current ceph status?
Thanks, Eugen
Zitat von Rok Jaklič <rjaklic@gmail.com>:
Hi,
we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything seems to work, however after some time radosgw stops responding and we have to restart it.
At first look, it seems that radosgw stops responding sometimes during recovery.
Does this maybe have to do something with mclock https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ?
ceph.conf looks something like this:
--------
[global] fsid = ... mon initial members = x1,x2 mon host = # public network = 192.168.0.0/24 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc
osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd crush chooseleaf type = 1
[osd] osd_scrub_begin_hour = 18 osd_scrub_end_hour = 6 osd_class_update_on_start = false osd_scrub_during_recovery = false #scrub during recovery osd_scrub_max_interval = 1209600 osd_deep_scrub_interval = 1209600 osd_max_scrubs = 3 osd_scrub_load_threshold = 1
[client.radosgw.mon2] host = mon2 # keyring = /etc/ceph/ceph.client.radosgw.keyring log_file = /var/log/ceph/client.radosgw.mon2.log rgw_dns_name = ... rgw_frontends = "beast port=4444" rgw_max_put_param_size = 15728640 rgw_crypt_require_ssl = false rgw_max_concurrent_requests = 2048 --------
We have nginx in front of rgw which upstream is set to client.radosgw.mon2 port 4444.
Kind regards, Rok _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
Rok Jaklič