ceph.log using seconds since epoch instead of date/time stamp
Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps? Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok # ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1 This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04. I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not): https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.1... Thanks, Bryan
I looked at that last year, I'm sure this is fixed in 17.2.8 and newer. Zitat von "Stillwell, Bryan" <bstillwe@akamai.com>:
Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps?
Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok
# ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1
This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04.
I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not):
https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.1...
Thanks, Bryan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen & Bryan, I've been trying to understand this issue -- I can't find anything in 17.2.8 that fixed it. Should we just assume that by switching the base image to el9 from el8 fixed it? Do we need to create a ticket here to look into this further? Cheers, Dan On Fri, Feb 21, 2025 at 9:44 AM Eugen Block <eblock@nde.ag> wrote:
I looked at that last year, I'm sure this is fixed in 17.2.8 and newer.
Zitat von "Stillwell, Bryan" <bstillwe@akamai.com>:
Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps?
Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok
# ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1
This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04.
I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not):
https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.1...
Thanks, Bryan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
Hi Dan, I just found my notes from last year, unfortunately without any "evidence". So I checked one of my lab clusters running with 17.2.8 (so it's el9), but the ceph.log is still using the epoch timestamp: quincy-1:~ # tail /var/log/ceph/1e6e5cb6-73e8-11ee-b195-fa163ee43e22/ceph.log 1743524881.6342404 mgr.quincy-1.pcasdd (mgr.2614124) 65 : cluster [DBG] pgmap v58: 361 pgs: 361 active+clean; 4.9 GiB data, 17 GiB used, 53 GiB / 70 GiB avail; 102 B/s rd, 0 op/s Note that I have log_to_file enabled, not sure how much a difference that makes. And I don't recall what my assumption was based on, but apparently it's not correct. I thought I had found a tracker last year, pointing to a pull request, but as I said, I didn't write down much. I'll keep looking if I find anything else. Zitat von Dan van der Ster <dan.vanderster@clyso.com>:
Hi Eugen & Bryan,
I've been trying to understand this issue -- I can't find anything in 17.2.8 that fixed it. Should we just assume that by switching the base image to el9 from el8 fixed it? Do we need to create a ticket here to look into this further?
Cheers, Dan
On Fri, Feb 21, 2025 at 9:44 AM Eugen Block <eblock@nde.ag> wrote:
I looked at that last year, I'm sure this is fixed in 17.2.8 and newer.
Zitat von "Stillwell, Bryan" <bstillwe@akamai.com>:
Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps?
Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok
# ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1
This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04.
I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not):
https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.1...
Thanks, Bryan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
Just to confirm, I redeployed my single-node test cluster with different versions. The logs are fine in: 17.2.6 (CentOS 8) 18.2.0 (CentOS 8) and further During the upgrade from 17.2.6 to 17.2.7 I see the log format switch after the MON has been upgraded: 2025-04-07T09:34:36.649854+0000 mgr.soc9-ceph.fjwsoh (mgr.14196) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 1744018480.121015 mon.soc9-ceph (mon.0) 1 : cluster [INF] mon.soc9-ceph is new leader, mons soc9-ceph in quorum (ranks 0) 1744018480.1211455 mon.soc9-ceph (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {soc9-ceph=[v2:192.168.124.186:3300/0,v1:192.168.124.186:6789/0]} removed_ranks: {} Upgrading it to 18.2.0 fixes it again. The only PR related to logs in the Quincy 17.2.7 changelog was this: https://github.com/ceph/ceph/pull/50777. But this is just a suspicion, I don't have anything to really confirm that. But it appears that it's not the container image version (CentOS 8). Zitat von Eugen Block <eblock@nde.ag>:
Hi Dan,
I just found my notes from last year, unfortunately without any "evidence". So I checked one of my lab clusters running with 17.2.8 (so it's el9), but the ceph.log is still using the epoch timestamp:
quincy-1:~ # tail /var/log/ceph/1e6e5cb6-73e8-11ee-b195-fa163ee43e22/ceph.log 1743524881.6342404 mgr.quincy-1.pcasdd (mgr.2614124) 65 : cluster [DBG] pgmap v58: 361 pgs: 361 active+clean; 4.9 GiB data, 17 GiB used, 53 GiB / 70 GiB avail; 102 B/s rd, 0 op/s
Note that I have log_to_file enabled, not sure how much a difference that makes. And I don't recall what my assumption was based on, but apparently it's not correct. I thought I had found a tracker last year, pointing to a pull request, but as I said, I didn't write down much. I'll keep looking if I find anything else.
Zitat von Dan van der Ster <dan.vanderster@clyso.com>:
Hi Eugen & Bryan,
I've been trying to understand this issue -- I can't find anything in 17.2.8 that fixed it. Should we just assume that by switching the base image to el9 from el8 fixed it? Do we need to create a ticket here to look into this further?
Cheers, Dan
On Fri, Feb 21, 2025 at 9:44 AM Eugen Block <eblock@nde.ag> wrote:
I looked at that last year, I'm sure this is fixed in 17.2.8 and newer.
Zitat von "Stillwell, Bryan" <bstillwe@akamai.com>:
Does anyone know why log entries in ceph.log and ceph.audit.log would use seconds since epoch on Quincy 17.2.7/Ubuntu 20.04, but ceph-mon.*.log and ceph-osd.*.log use date/time stamps?
Example: # ceph.log 1740114003.665984 osd.243 (osd.243) 2137 : cluster [DBG] 3.1615 scrub ok
# ceph-mon.log 2025-02-21T00:00:01.814-0500 7fe33378e700 0 mon.XXXXXX@0(leader) e2 handle_command mon_command({"prefix": "osd dump"} v 0) v1
This doesn't seem to be the case on my home cluster running Squid 19.2.1/Ubuntu 24.04.
I found this Proxmox forum post that seems to imply that could be a distro setting (same Ceph version, but bookworm is affected, and bullseye is not):
https://forum.proxmox.com/threads/ceph-log-file-switched-to-unix-timestamp.1...
Thanks, Bryan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
participants (3)
-
Dan van der Ster
-
Eugen Block
-
Stillwell, Bryan