So it seems some ceph housekeeping spawn containers without giving them a name and that causes this in the journal: " Feb 01 04:10:07 dopey podman[766731]: 2024-02-01 04:10:07.967786606 +0100 CET m=+0.043987882 container create 95967a040795bd61588dcfdc6ba5daf92553cd2cb3ecd7318cd8b16c1b15782d (image=quay.io/ceph/ceph@sha256:1793ff3af6ae74527c86e1a0b22401e9c42dc08d0ebb8379653be07db17d0007, name=practical_hypatia, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, ceph=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.0, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) ... Feb 01 04:10:08 dopey practical_hypatia[766758]: 167 167 ... Feb 01 04:10:08 dopey systemd[1]: libpod-conmon-95967a040795bd61588dcfdc6ba5daf92553cd2cb3ecd7318cd8b16c1b15782d.scope: Deactivated successfully " Mvh. Torkil On 01/02/2024 08:24, Torkil Svensgaard wrote:
We have ceph (currently 18.2.0) log to an rsyslog server with the following file name format:
template (name="DynaFile" type="string" string="/tank/syslog/%fromhost-ip%/%hostname%/%programname%.log")
Around May 25th this year something changed so instead of getting the usual program log names we are also getting a lot of diffent logs with weird names. Here's an ls excerpt:
" ... -rw-------. 1 root root 517K Feb 1 05:42 hardcore_raman.log -rw-------. 1 root root 198K Feb 1 05:42 sleepy_moser.log -rw-------. 1 root root 203K Feb 1 05:42 friendly_gagarin.log -rw-------. 1 root root 1.1K Feb 1 05:42 goofy_hypatia.log -rw-------. 1 root root 164K Feb 1 05:42 kind_chebyshev.log -rw-------. 1 root root 11K Feb 1 05:42 magical_archimedes.log -rw-------. 1 root root 373K Feb 1 05:42 busy_bardeen.log -rw-------. 1 root root 178K Feb 1 05:42 trusting_euler.log -rw-------. 1 root root 526K Feb 1 05:42 inspiring_golick.log -rw-------. 1 root root 369K Feb 1 06:12 condescending_ganguly.log -rw-------. 1 root root 191K Feb 1 06:12 mystifying_torvalds.log -rw-------. 1 root root 475K Feb 1 06:12 charming_nash.log -rw-------. 1 root root 168K Feb 1 06:12 zealous_sinoussi.log -rw-------. 1 root root 325K Feb 1 06:12 amazing_booth.log -rw-------. 1 root root 516K Feb 1 06:12 great_ardinghelli.log -rw-------. 1 root root 313K Feb 1 06:12 magical_bell.log -rw-------. 1 root root 22K Feb 1 06:12 nifty_swartz.log -rw-------. 1 root root 426 Feb 1 06:12 upbeat_beaver.log -rw-------. 1 root root 166K Feb 1 06:13 funny_lederberg.log -rw-------. 1 root root 164K Feb 1 06:13 frosty_murdock.log -rw-------. 1 root root 374K Feb 1 06:13 elastic_banach.log -rw-------. 1 root root 308K Feb 1 06:13 inspiring_cohen.log -rw-------. 1 root root 176K Feb 1 06:13 angry_wu.log -rw-------. 1 root root 662 Feb 1 06:42 admiring_kalam.log -rw-------. 1 root root 3.1K Feb 1 06:43 thirsty_colden.log -rw-------. 1 root root 4.5M Feb 1 07:01 run-parts.log -rw-------. 1 root root 16M Feb 1 07:01 CROND.log -rw-------. 1 root root 109M Feb 1 07:06 python3.log -rw-------. 1 root root 3.4M Feb 1 07:29 systemd-journald.log -rw-------. 1 root root 596M Feb 1 07:34 sudo.log -rw-------. 1 root root 549 Feb 1 07:44 interesting_rosalind.log -rw-------. 1 root root 342K Feb 1 07:45 beautiful_hamilton.log -rw-------. 1 root root 348K Feb 1 07:45 cool_ride.log -rw-------. 1 root root 15G Feb 1 07:45 conmon.log -rw-------. 1 root root 395K Feb 1 07:45 compassionate_satoshi.log -rw-------. 1 root root 11K Feb 1 07:45 hardcore_noether.log -rw-------. 1 root root 223K Feb 1 07:45 wizardly_johnson.log -rw-------. 1 root root 270M Feb 1 07:49 sshd.log -rw-------. 1 root root 111M Feb 1 07:49 systemd-logind.log -rw-------. 1 root root 1.6G Feb 1 07:50 systemd.log -rw-------. 1 root root 119M Feb 1 07:54 rsyslogd.log -rw-------. 1 root root 94G Feb 1 07:55 ceph-8ee2d228-ed21-4580-8bbf-064.log -rw-------. 1 root root 1.1G Feb 1 07:56 podman.log -rw-------. 1 root root 1.8G Feb 1 07:58 ceph-mgr.log -rw-------. 1 root root 213G Feb 1 07:58 ceph-osd.log -rw-------. 1 root root 48G Feb 1 07:58 ceph-mon.log
"
Those are container names or something like that? The file content seems to be assorted bits from the ceph disk tool:
" # cat goofy_hypatia.log Jun 7 04:12:43 dopey goofy_hypatia[3224681]: 167 167 Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> passed data devices: 22 physical, 0 LVM Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> relative data size: 1.0 Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> All data devices are unavailable Jun 24 09:00:08 dopey goofy_hypatia[2319188]: [] Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> passed data devices: 22 physical, 0 LVM Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> relative data size: 1.0 Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> All data devices are unavailable 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> passed data devices: 2 physical, 0 LVM 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> relative data size: 1.0 2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> All data devices are unavailable 2023-12-26T09:38:59+01:00 dopey goofy_hypatia[893177]: 167 167 2023-12-26T10:41:14+01:00 dopey goofy_hypatia[1057467]: 167 167 2024-01-05T21:00:37+01:00 dopey goofy_hypatia[3395314]: {} 2024-02-01T05:42:17+01:00 dopey goofy_hypatia[845150]: 167 167 "
Anyone else had this issue? Suggestions on how to get a real program name instead?
Mvh.
Torkil
-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk