Reef: RGW logrotate "breaks" log_to_file
Hi, after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate. I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate. Is this a known issue? I can't seem to find anything on tracker. Thanks! Eugen ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
Hi, I would recommend to upgrade to 18.2.7. Regards Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Eugen Block <eblock@nde.ag> schrieb am Mi., 25. Juni 2025, 10:05:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry, I should have mentioned that those clusters are all on 18.2.7 already, I just wanted to state that the behavior was introduced in 18.2.5. Zitat von Joachim Kraftmayer <joachim.kraftmayer@clyso.com>:
Hi, I would recommend to upgrade to 18.2.7.
Regards Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Eugen Block <eblock@nde.ag> schrieb am Mi., 25. Juni 2025, 10:05:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade? Zitat von Eugen Block <eblock@nde.ag>:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
hi Eugen, this is tracked in https://tracker.ceph.com/issues/71156, and a fix has merged for the upcoming 19.2.3 release unfortunately, a mistake was made in backporting some changes related to thread names and the radosgw process gets renamed to "notif-worker0" as a result. so commands like pkill expect that string instead of radosgw On Wed, Jun 25, 2025 at 7:00 AM Eugen Block <eblock@nde.ag> wrote:
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade?
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Jun 25, 2025 at 8:29 AM Casey Bodley <cbodley@redhat.com> wrote:
hi Eugen,
this is tracked in https://tracker.ceph.com/issues/71156, and a fix has merged for the upcoming 19.2.3 release
oops, i said 19.2.3 but meant the next reef
unfortunately, a mistake was made in backporting some changes related to thread names and the radosgw process gets renamed to "notif-worker0" as a result. so commands like pkill expect that string instead of radosgw
On Wed, Jun 25, 2025 at 7:00 AM Eugen Block <eblock@nde.ag> wrote:
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade?
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is there any negative to adding notif-worker0 to the logrotate pkill list? On Wed, Jun 25, 2025 at 8:32 AM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jun 25, 2025 at 8:29 AM Casey Bodley <cbodley@redhat.com> wrote:
hi Eugen,
this is tracked in https://tracker.ceph.com/issues/71156, and a fix has merged for the upcoming 19.2.3 release
oops, i said 19.2.3 but meant the next reef
unfortunately, a mistake was made in backporting some changes related to thread names and the radosgw process gets renamed to "notif-worker0" as a result. so commands like pkill expect that string instead of radosgw
On Wed, Jun 25, 2025 at 7:00 AM Eugen Block <eblock@nde.ag> wrote:
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade?
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks a lot, Casey. I'm still not sure why I couldn't find that myself, but thanks anyway. I have added notif-worker0 to the logrotate file in both a test cluster and one production cluster, pkill on that process does trigger an entry in the log file (after restarting the rgw daemon), so tomorrow morning I will see if that's a workaround until the next upgrade. Thanks again for the quick response! Eugen Zitat von Reid Guyett <reid.guyett@gmail.com>:
Is there any negative to adding notif-worker0 to the logrotate pkill list?
On Wed, Jun 25, 2025 at 8:32 AM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jun 25, 2025 at 8:29 AM Casey Bodley <cbodley@redhat.com> wrote:
hi Eugen,
this is tracked in https://tracker.ceph.com/issues/71156, and a fix has merged for the upcoming 19.2.3 release
oops, i said 19.2.3 but meant the next reef
unfortunately, a mistake was made in backporting some changes related to thread names and the radosgw process gets renamed to "notif-worker0" as a result. so commands like pkill expect that string instead of radosgw
On Wed, Jun 25, 2025 at 7:00 AM Eugen Block <eblock@nde.ag> wrote:
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade?
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, that worked as expected. I can't see any negative impact yet. Zitat von Eugen Block <eblock@nde.ag>:
Thanks a lot, Casey. I'm still not sure why I couldn't find that myself, but thanks anyway. I have added notif-worker0 to the logrotate file in both a test cluster and one production cluster, pkill on that process does trigger an entry in the log file (after restarting the rgw daemon), so tomorrow morning I will see if that's a workaround until the next upgrade.
Thanks again for the quick response! Eugen
Zitat von Reid Guyett <reid.guyett@gmail.com>:
Is there any negative to adding notif-worker0 to the logrotate pkill list?
On Wed, Jun 25, 2025 at 8:32 AM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jun 25, 2025 at 8:29 AM Casey Bodley <cbodley@redhat.com> wrote:
hi Eugen,
this is tracked in https://tracker.ceph.com/issues/71156, and a fix has merged for the upcoming 19.2.3 release
oops, i said 19.2.3 but meant the next reef
unfortunately, a mistake was made in backporting some changes related to thread names and the radosgw process gets renamed to "notif-worker0" as a result. so commands like pkill expect that string instead of radosgw
On Wed, Jun 25, 2025 at 7:00 AM Eugen Block <eblock@nde.ag> wrote:
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade?
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You might want to be careful when running radosgw-admin commands that could take a long time to finish (like radosgw-admin bucket rm), because these will also be killed since these match the notif-worker0 basename. Arthur
Thanks a lot, Casey. I'm still not sure why I couldn't find that myself, but thanks anyway. I have added notif-worker0 to the logrotate file in both a test cluster and one production cluster, pkill on that process does trigger an entry in the log file (after restarting the rgw daemon), so tomorrow morning I will see if that's a workaround until the next upgrade.
Thanks again for the quick response! Eugen
Zitat von Reid Guyett <reid.guyett@gmail.com>:
Is there any negative to adding notif-worker0 to the logrotate pkill list?
On Wed, Jun 25, 2025 at 8:32 AM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jun 25, 2025 at 8:29 AM Casey Bodley <cbodley@redhat.com> wrote:
hi Eugen,
this is tracked in https://tracker.ceph.com/issues/71156, and a fix has merged for the upcoming 19.2.3 release
oops, i said 19.2.3 but meant the next reef
unfortunately, a mistake was made in backporting some changes related to thread names and the radosgw process gets renamed to "notif-worker0" as a result. so commands like pkill expect that string instead of radosgw
On Wed, Jun 25, 2025 at 7:00 AM Eugen Block <eblock@nde.ag> wrote:
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade?
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate.
I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate.
Is this a known issue? I can't seem to find anything on tracker.
Thanks! Eugen
ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the info, that makes sense. ALthough I'm usually the only person running commands like those on those clusters, so I should be safe. :-) Zitat von Arthur van Kleef <arthur.v.kleef@belsimpel.nl>:
You might want to be careful when running radosgw-admin commands that could take a long time to finish (like radosgw-admin bucket rm), because these will also be killed since these match the notif-worker0 basename.
Arthur
Thanks a lot, Casey. I'm still not sure why I couldn't find that myself, but thanks anyway. I have added notif-worker0 to the logrotate file in both a test cluster and one production cluster, pkill on that process does trigger an entry in the log file (after restarting the rgw daemon), so tomorrow morning I will see if that's a workaround until the next upgrade.
Thanks again for the quick response! Eugen
Zitat von Reid Guyett <reid.guyett@gmail.com>:
Is there any negative to adding notif-worker0 to the logrotate pkill list?
On Wed, Jun 25, 2025 at 8:32 AM Casey Bodley <cbodley@redhat.com> wrote:
On Wed, Jun 25, 2025 at 8:29 AM Casey Bodley <cbodley@redhat.com> wrote:
hi Eugen,
this is tracked in https://tracker.ceph.com/issues/71156, and a fix has merged for the upcoming 19.2.3 release
oops, i said 19.2.3 but meant the next reef
unfortunately, a mistake was made in backporting some changes related to thread names and the radosgw process gets renamed to "notif-worker0" as a result. so commands like pkill expect that string instead of radosgw
On Wed, Jun 25, 2025 at 7:00 AM Eugen Block <eblock@nde.ag> wrote:
Interesting, it seems like the pkill command from logrotate doesn't trigger anything on the RGW daemons in Reef >= 18.2.5. But on 19.2.2 it does work as expected. Right now an upgrade to Squid is not possible, and this has the unfortunate side effect of blowing up the file system until the process is restarted. Is there some workaround possible until we upgrade?
Zitat von Eugen Block <eblock@nde.ag>:
> Hi, > > after upgrading multiple clusters from 18.2.4. some weeks ago, I > noticed that the RGWs stop logging to file after the nightly > logrotate. Other daemons don't seem to be affected, they continue > logging to file. Restarting an RGW daemon helps until the next > logrotate. > > I could reproduce this in a lab cluster, after upgrading from 18.2.4 > to 18.2.5, the daemon stops logging to file after a logrotate. > > Is this a known issue? I can't seem to find anything on tracker. > > Thanks! > Eugen > > ceph config dump | grep log_to_file > global basic > log_to_file true > global advanced > mon_cluster_log_to_file true
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Arthur van Kleef
-
Casey Bodley
-
Eugen Block
-
Joachim Kraftmayer
-
Reid Guyett