cephfs filesystem offline, ceph-mds core dumped... How to recover?
Hi All, We just noticed that our cephfs filesystem was unreachable and we saw this: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr [root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more And filesystem status is: # ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there: # tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225254: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225255: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225256: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:26] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225257: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225258: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225259: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225260: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225261: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:36] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225262: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225263: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225264: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225265: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225266: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:46] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:06:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225267: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225268: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225269: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225270: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225271: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:56] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225272: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225273: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228438. Immutable memtables: 0. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.708981) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12100] Flushing memtable with next log file: 228438 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819709068, "job": 12100, "event": "flush_started", "num_memtables": 1, "num_entries": 3587, "num_deletes": 921, "total_data_size": 2910092, "memory_usage": 2992096, "flush_reason": "Manual Compaction"} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12100] Level-0 flush table #228439: started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819721834, "cf_name": "default", "job": 12100, "event": "table_file_creation", "file_number": 228439, "file_size": 1847607, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141701983, "largest_seqno": 141705565, "table_properties": {"data_size": 1834448, "index_size": 6860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5253, "raw_key_size": 55412, "raw_average_key_size": 26, "raw_value_size": 1798552, "raw_average_value_size": 857, "num_data_blocks": 265, "num_entries": 2097, "num_filter_entries": 2097, "num_deletions": 921, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126677, "oldest_key_time": 1715126677, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228439, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12100] Flush lasted 12882 microseconds, and 5541 cpu microseconds. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721870) [db/flush_job.cc:967] [default] [JOB 12100] Level-0 flush table #228439: 1847607 bytes OK May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721889) [db/memtable_list.cc:519] [default] Level-0 commit table #228439 started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724483) [db/memtable_list.cc:722] [default] Level-0 commit table #228439: memtable #1 done May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724523) EVENT_LOG_v1 {"time_micros": 1715126819724510, "job": 12100, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724561) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12100] Try to delete WAL files size 2890782, prev total WAL file size 2890782, number of live WAL files 2. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725271) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337383930' seq:72057594037927935, type:22 .. '7061786F73003130313338313432' seq:0, type:0; will stop at (end) May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12101] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12100 Base level 0, inputs: [228439(1804KB)], [228437(36MB)] May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819725328, "job": 12101, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228439], "files_L6": [228437], "score": -1, "input_data_size": 39995951, "oldest_snapshot_seqno": -1} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12101] Generated table #228440: 25379 keys, 38251292 bytes, temperature: kUnknown May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819875843, "cf_name": "default", "job": 12101, "event": "table_file_creation", "file_number": 228440, "file_size": 38251292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 38096937, "index_size": 89812, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702456, "raw_average_key_size": 27, "raw_value_size": 37648320, "raw_average_value_size": 1483, "num_data_blocks": 3163, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228440, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.876095) [db/compaction/compaction_job.cc:1663] [default] [JOB 12101] Compacted 1@0 + 1@6 files to L6 => 38251292 bytes May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878629) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 265.6 rd, 254.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.8, 36.4 +0.0 blob) out(36.5 +0.0 blob), read-write-amplify(42.4) write-amplify(20.7) OK, records in: 27250, records dropped: 1871 output_compression: NoCompression May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878669) EVENT_LOG_v1 {"time_micros": 1715126819878658, "job": 12101, "event": "compaction_finished", "compaction_time_micros": 150577, "compaction_time_cpu_micros": 64114, "output_level": 6, "num_output_files": 1, "total_output_size": 38251292, "num_input_records": 27250, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819878937, "job": 12101, "event": "table_file_deletion", "file_number": 228439} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819882652, "job": 12101, "event": "table_file_deletion", "file_number": 228437} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882737) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882741) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882743) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882744) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882746) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:07:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225274: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225275: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225276: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:06] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:07:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225277: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:07:07 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_data', 'cephfs_md_and_data', '.mgr', 'cephfs_metadata'] May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:07:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:07:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225278: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225279: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225280: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:07:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225281: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 e5614: 46 total, 46 up, 46 in May 7 17:07:15 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225283: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d807ba79-465d-4d89-ab03-9bff6275dd10 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9e26e8f6-fba9-425c-923b-ba9da32693a7 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1f435daa-e8f0-4964-b7fb-07ea14b95e9f does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.694558647 -0700 PDT m=+0.039788036 container create 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.722272344 -0700 PDT m=+0.067501679 container init 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:07:17 pr-md-02 stoic_mayer[2565752]: 167 167 May 7 17:07:17 pr-md-02 systemd[1]: libpod-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728419093 -0700 PDT m=+0.073648421 container start 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728522035 -0700 PDT m=+0.073751360 container attach 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728769341 -0700 PDT m=+0.073998662 container died 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.678321769 -0700 PDT m=+0.023551090 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:17 pr-md-02 podman[2565757]: 2024-05-07 17:07:17.78259101 -0700 PDT m=+0.046708660 container remove 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 systemd[1]: libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.90851263 -0700 PDT m=+0.039669496 container create 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.936966849 -0700 PDT m=+0.068123675 container init 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943702066 -0700 PDT m=+0.074858891 container start 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943794994 -0700 PDT m=+0.074951827 container attach 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.892929482 -0700 PDT m=+0.024086307 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> passed data devices: 2 physical, 0 LVM May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> relative data size: 1.0 May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> All data devices are unavailable May 7 17:07:18 pr-md-02 systemd[1]: libpod-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 podman[2565778]: 2024-05-07 17:07:18.556739188 -0700 PDT m=+0.687896012 container died 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:18 pr-md-02 podman[2565822]: 2024-05-07 17:07:18.614711286 -0700 PDT m=+0.047414226 container remove 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:18 pr-md-02 systemd[1]: libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-2f63fd9ffd2763a7ee3f75befaf99b27a75325cd025cc2e74e72d7bd150afc42-merged.mount: Deactivated successfully. May 7 17:07:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225284: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.171860525 -0700 PDT m=+0.039261474 container create eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.199519439 -0700 PDT m=+0.066920407 container init eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:19 pr-md-02 xenodochial_wu[2565974]: 167 167 May 7 17:07:19 pr-md-02 systemd[1]: libpod-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.205924433 -0700 PDT m=+0.073325392 container start eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.2060646 -0700 PDT m=+0.073465559 container attach eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.206298065 -0700 PDT m=+0.073699031 container died eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.158156321 -0700 PDT m=+0.025557273 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 podman[2565979]: 2024-05-07 17:07:19.25994464 -0700 PDT m=+0.046894141 container remove eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.388083595 -0700 PDT m=+0.043236647 container create a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.417053374 -0700 PDT m=+0.072206356 container init a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423715752 -0700 PDT m=+0.078868733 container start a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423810643 -0700 PDT m=+0.078963634 container attach a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.371099332 -0700 PDT m=+0.026252317 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-91f5899d00d5794e67bbdd45fa72d22c74b0b977efec0238aa842ac272b635ce-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "41": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme0n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "41", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "43": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme1n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "43", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ] May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 systemd[1]: libpod-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.801695942 -0700 PDT m=+0.456848932 container died a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-68bc380ea9ec1cff282d08f862c7b661ba0571a20622225a5a3994e5fc88985d-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566028]: 2024-05-07 17:07:19.854983202 -0700 PDT m=+0.045986655 container remove a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.420915201 -0700 PDT m=+0.041065865 container create f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.45143613 -0700 PDT m=+0.071586769 container init f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:07:20 pr-md-02 practical_wu[2566180]: 167 167 May 7 17:07:20 pr-md-02 systemd[1]: libpod-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457513493 -0700 PDT m=+0.077664122 container start f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457638676 -0700 PDT m=+0.077789308 container attach f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457913249 -0700 PDT m=+0.078063874 container died f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.405495878 -0700 PDT m=+0.025646501 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 podman[2566185]: 2024-05-07 17:07:20.515553442 -0700 PDT m=+0.045837844 container remove f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) May 7 17:07:20 pr-md-02 systemd[1]: libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.631476902 -0700 PDT m=+0.043759539 container create c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.660235834 -0700 PDT m=+0.072518401 container init c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.666915701 -0700 PDT m=+0.079198260 container start c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.667002215 -0700 PDT m=+0.079284787 container attach c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) May 7 17:07:20 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb2679257a24186947b435237af458658d281389b531640688ba39cfa7cac79-merged.mount: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.6134279 -0700 PDT m=+0.025710470 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225285: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 43, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: }, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 41, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 systemd[1]: libpod-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566207]: 2024-05-07 17:07:21.655826716 -0700 PDT m=+1.068109291 container died c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:21 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-e573f9cf164334d9b913d82ba760fa7f00e98646ea067755c99a3842969d30d8-merged.mount: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566303]: 2024-05-07 17:07:21.711419562 -0700 PDT m=+0.047850993 container remove c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:21 pr-md-02 systemd[1]: libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 57f8febe-215e-403c-821d-95da6e3c5073 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 2637522f-3bd6-453c-976c-f7da78d12399 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b0cc7927-d150-4870-8ec7-6ef7d847ffd2 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9b053170-587d-42ee-929e-25112b3abba1 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 352cfbfb-4b0a-420c-959b-e42615f9913d does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 144dbb62-1eeb-49ba-b928-eee4a1756251 does not exist May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225286: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225287: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225288: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:29 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225289: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:31 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225290: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:33 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225291: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:35 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225292: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:37 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225293: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:39 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225294: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:41 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225295: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:43 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225296: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:45 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225297: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:46] "GET /metrics HTTP/1.1" 200 73538 "" "Prometheus/2.43.0" May 7 17:07:47 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225298: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:49 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225299: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:51 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225300: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:53 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225301: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225302: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:56] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:07:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225303: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:59 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225304: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225305: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225306: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:04 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225307: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub starts May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub ok May 7 17:08:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:06] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:08:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225308: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 e5615: 46 total, 46 up, 46 in May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:08:07 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', 'cephfs_data', '.mgr', 'cephfs_metadata'] May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:08:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:08:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225310: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:09 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:08:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225311: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225312: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:08:14 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225313: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:16] "GET /metrics HTTP/1.1" 200 73536 "" "Prometheus/2.43.0" May 7 17:08:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225314: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228441. Immutable memtables: 0. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.105603) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12102] Flushing memtable with next log file: 228441 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898105644, "job": 12102, "event": "flush_started", "num_memtables": 1, "num_entries": 2046, "num_deletes": 617, "total_data_size": 1704550, "memory_usage": 1744008, "flush_reason": "Manual Compaction"} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12102] Level-0 flush table #228442: started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898113765, "cf_name": "default", "job": 12102, "event": "table_file_creation", "file_number": 228442, "file_size": 847046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141705570, "largest_seqno": 141707611, "table_properties": {"data_size": 839933, "index_size": 2991, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 32187, "raw_average_key_size": 26, "raw_value_size": 819966, "raw_average_value_size": 670, "num_data_blocks": 115, "num_entries": 1223, "num_filter_entries": 1223, "num_deletions": 617, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126820, "oldest_key_time": 1715126820, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228442, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12102] Flush lasted 8207 microseconds, and 2745 cpu microseconds. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113810) [db/flush_job.cc:967] [default] [JOB 12102] Level-0 flush table #228442: 847046 bytes OK May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113830) [db/memtable_list.cc:519] [default] Level-0 commit table #228442 started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116583) [db/memtable_list.cc:722] [default] Level-0 commit table #228442: memtable #1 done May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116634) EVENT_LOG_v1 {"time_micros": 1715126898116612, "job": 12102, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116650) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12102] Try to delete WAL files size 1693223, prev total WAL file size 1693550, number of live WAL files 2. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228438.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.117035) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740032393030383037' seq:72057594037927935, type:22 .. '6D6772737461740032393031303630' seq:0, type:0; will stop at (end) May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12103] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12102 Base level 0, inputs: [228442(827KB)], [228440(36MB)] May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898117062, "job": 12103, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228442], "files_L6": [228440], "score": -1, "input_data_size": 39098338, "oldest_snapshot_seqno": -1} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12103] Generated table #228443: 25379 keys, 37258556 bytes, temperature: kUnknown May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898320688, "cf_name": "default", "job": 12103, "event": "table_file_creation", "file_number": 228443, "file_size": 37258556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37107602, "index_size": 86411, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702042, "raw_average_key_size": 27, "raw_value_size": 36662378, "raw_average_value_size": 1444, "num_data_blocks": 3038, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228443, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.321010) [db/compaction/compaction_job.cc:1663] [default] [JOB 12103] Compacted 1@0 + 1@6 files to L6 => 37258556 bytes May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323275) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 191.9 rd, 182.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.8, 36.5 +0.0 blob) out(35.5 +0.0 blob), read-write-amplify(90.1) write-amplify(44.0) OK, records in: 26602, records dropped: 1223 output_compression: NoCompression May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323329) EVENT_LOG_v1 {"time_micros": 1715126898323308, "job": 12103, "event": "compaction_finished", "compaction_time_micros": 203764, "compaction_time_cpu_micros": 65018, "output_level": 6, "num_output_files": 1, "total_output_size": 37258556, "num_input_records": 26602, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898323660, "job": 12103, "event": "table_file_deletion", "file_number": 228442} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898326958, "job": 12103, "event": "table_file_deletion", "file_number": 228440} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116998) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326990) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326994) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327002) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327006) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327008) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting I have not updated any packages or anything, I'm not sure how this came about. I did set the following two items earlier today, although I'm not sure if it would make any difference? ceph config set mds mds_recall_max_caps 3000 ceph config set mds mds_cache_memory_limit 16000000000 I tried to restart the MDS daemons alike so: ceph orch daemon restart mds.slugfs.pr-md-02.sbblqq And they seem to start for a moment then stop again. I then tried to restart the one remaining one that was online, then it seemed to die as well, and now I see this: # ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) But the ceph-mds on pr-md-01 doesn't exist now. Anyone know how to recover from this? I'm kind of panicking. :( Thanks much, erich
Actually looking again, I don't see core dumps (though I could have sworn I did before). But looking at my MDS servers, they appear to be in a constant cycle of trying to restart, and not making it, then trying again, etc. The logs look like what I sent in my previous email, over and over again. I tried restarting all three of my ceph-mds daemons at the same time, and it still doesn't seem to help. I hope I can recover... -erich On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,
We just noticed that our cephfs filesystem was unreachable and we saw this:
# ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w)
data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep
io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr
[root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more
And filesystem status is:
# ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there:
# tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225254: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225255: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225256: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:26] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225257: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225258: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225259: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225260: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225261: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:36] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225262: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225263: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225264: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225265: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225266: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:46] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:06:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225267: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225268: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225269: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225270: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225271: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:56] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225272: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225273: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228438. Immutable memtables: 0. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.708981) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12100] Flushing memtable with next log file: 228438 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819709068, "job": 12100, "event": "flush_started", "num_memtables": 1, "num_entries": 3587, "num_deletes": 921, "total_data_size": 2910092, "memory_usage": 2992096, "flush_reason": "Manual Compaction"} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12100] Level-0 flush table #228439: started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819721834, "cf_name": "default", "job": 12100, "event": "table_file_creation", "file_number": 228439, "file_size": 1847607, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141701983, "largest_seqno": 141705565, "table_properties": {"data_size": 1834448, "index_size": 6860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5253, "raw_key_size": 55412, "raw_average_key_size": 26, "raw_value_size": 1798552, "raw_average_value_size": 857, "num_data_blocks": 265, "num_entries": 2097, "num_filter_entries": 2097, "num_deletions": 921, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126677, "oldest_key_time": 1715126677, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228439, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12100] Flush lasted 12882 microseconds, and 5541 cpu microseconds. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721870) [db/flush_job.cc:967] [default] [JOB 12100] Level-0 flush table #228439: 1847607 bytes OK May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721889) [db/memtable_list.cc:519] [default] Level-0 commit table #228439 started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724483) [db/memtable_list.cc:722] [default] Level-0 commit table #228439: memtable #1 done May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724523) EVENT_LOG_v1 {"time_micros": 1715126819724510, "job": 12100, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724561) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12100] Try to delete WAL files size 2890782, prev total WAL file size 2890782, number of live WAL files 2. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725271) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337383930' seq:72057594037927935, type:22 .. '7061786F73003130313338313432' seq:0, type:0; will stop at (end) May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12101] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12100 Base level 0, inputs: [228439(1804KB)], [228437(36MB)] May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819725328, "job": 12101, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228439], "files_L6": [228437], "score": -1, "input_data_size": 39995951, "oldest_snapshot_seqno": -1} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12101] Generated table #228440: 25379 keys, 38251292 bytes, temperature: kUnknown May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819875843, "cf_name": "default", "job": 12101, "event": "table_file_creation", "file_number": 228440, "file_size": 38251292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 38096937, "index_size": 89812, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702456, "raw_average_key_size": 27, "raw_value_size": 37648320, "raw_average_value_size": 1483, "num_data_blocks": 3163, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228440, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.876095) [db/compaction/compaction_job.cc:1663] [default] [JOB 12101] Compacted 1@0 + 1@6 files to L6 => 38251292 bytes May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878629) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 265.6 rd, 254.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.8, 36.4 +0.0 blob) out(36.5 +0.0 blob), read-write-amplify(42.4) write-amplify(20.7) OK, records in: 27250, records dropped: 1871 output_compression: NoCompression May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878669) EVENT_LOG_v1 {"time_micros": 1715126819878658, "job": 12101, "event": "compaction_finished", "compaction_time_micros": 150577, "compaction_time_cpu_micros": 64114, "output_level": 6, "num_output_files": 1, "total_output_size": 38251292, "num_input_records": 27250, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819878937, "job": 12101, "event": "table_file_deletion", "file_number": 228439} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819882652, "job": 12101, "event": "table_file_deletion", "file_number": 228437} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882737) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882741) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882743) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882744) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882746) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:07:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225274: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225275: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225276: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:06] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:07:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225277: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:07:07 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_data', 'cephfs_md_and_data', '.mgr', 'cephfs_metadata'] May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:07:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:07:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225278: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225279: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225280: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:07:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225281: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 e5614: 46 total, 46 up, 46 in May 7 17:07:15 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225283: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d807ba79-465d-4d89-ab03-9bff6275dd10 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9e26e8f6-fba9-425c-923b-ba9da32693a7 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1f435daa-e8f0-4964-b7fb-07ea14b95e9f does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.694558647 -0700 PDT m=+0.039788036 container create 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.722272344 -0700 PDT m=+0.067501679 container init 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:07:17 pr-md-02 stoic_mayer[2565752]: 167 167 May 7 17:07:17 pr-md-02 systemd[1]: libpod-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728419093 -0700 PDT m=+0.073648421 container start 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728522035 -0700 PDT m=+0.073751360 container attach 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728769341 -0700 PDT m=+0.073998662 container died 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.678321769 -0700 PDT m=+0.023551090 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:17 pr-md-02 podman[2565757]: 2024-05-07 17:07:17.78259101 -0700 PDT m=+0.046708660 container remove 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 systemd[1]: libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.90851263 -0700 PDT m=+0.039669496 container create 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.936966849 -0700 PDT m=+0.068123675 container init 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943702066 -0700 PDT m=+0.074858891 container start 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943794994 -0700 PDT m=+0.074951827 container attach 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.892929482 -0700 PDT m=+0.024086307 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> passed data devices: 2 physical, 0 LVM May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> relative data size: 1.0 May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> All data devices are unavailable May 7 17:07:18 pr-md-02 systemd[1]: libpod-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 podman[2565778]: 2024-05-07 17:07:18.556739188 -0700 PDT m=+0.687896012 container died 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:18 pr-md-02 podman[2565822]: 2024-05-07 17:07:18.614711286 -0700 PDT m=+0.047414226 container remove 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:18 pr-md-02 systemd[1]: libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-2f63fd9ffd2763a7ee3f75befaf99b27a75325cd025cc2e74e72d7bd150afc42-merged.mount: Deactivated successfully. May 7 17:07:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225284: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.171860525 -0700 PDT m=+0.039261474 container create eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.199519439 -0700 PDT m=+0.066920407 container init eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:19 pr-md-02 xenodochial_wu[2565974]: 167 167 May 7 17:07:19 pr-md-02 systemd[1]: libpod-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.205924433 -0700 PDT m=+0.073325392 container start eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.2060646 -0700 PDT m=+0.073465559 container attach eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.206298065 -0700 PDT m=+0.073699031 container died eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.158156321 -0700 PDT m=+0.025557273 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 podman[2565979]: 2024-05-07 17:07:19.25994464 -0700 PDT m=+0.046894141 container remove eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.388083595 -0700 PDT m=+0.043236647 container create a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.417053374 -0700 PDT m=+0.072206356 container init a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423715752 -0700 PDT m=+0.078868733 container start a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423810643 -0700 PDT m=+0.078963634 container attach a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.371099332 -0700 PDT m=+0.026252317 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-91f5899d00d5794e67bbdd45fa72d22c74b0b977efec0238aa842ac272b635ce-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "41": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme0n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "41", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "43": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme1n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "43", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ] May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 systemd[1]: libpod-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.801695942 -0700 PDT m=+0.456848932 container died a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-68bc380ea9ec1cff282d08f862c7b661ba0571a20622225a5a3994e5fc88985d-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566028]: 2024-05-07 17:07:19.854983202 -0700 PDT m=+0.045986655 container remove a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.420915201 -0700 PDT m=+0.041065865 container create f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.45143613 -0700 PDT m=+0.071586769 container init f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:07:20 pr-md-02 practical_wu[2566180]: 167 167 May 7 17:07:20 pr-md-02 systemd[1]: libpod-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457513493 -0700 PDT m=+0.077664122 container start f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457638676 -0700 PDT m=+0.077789308 container attach f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457913249 -0700 PDT m=+0.078063874 container died f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.405495878 -0700 PDT m=+0.025646501 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 podman[2566185]: 2024-05-07 17:07:20.515553442 -0700 PDT m=+0.045837844 container remove f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) May 7 17:07:20 pr-md-02 systemd[1]: libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.631476902 -0700 PDT m=+0.043759539 container create c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.660235834 -0700 PDT m=+0.072518401 container init c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.666915701 -0700 PDT m=+0.079198260 container start c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.667002215 -0700 PDT m=+0.079284787 container attach c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) May 7 17:07:20 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb2679257a24186947b435237af458658d281389b531640688ba39cfa7cac79-merged.mount: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.6134279 -0700 PDT m=+0.025710470 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225285: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 43, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: }, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 41, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 systemd[1]: libpod-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566207]: 2024-05-07 17:07:21.655826716 -0700 PDT m=+1.068109291 container died c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:21 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-e573f9cf164334d9b913d82ba760fa7f00e98646ea067755c99a3842969d30d8-merged.mount: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566303]: 2024-05-07 17:07:21.711419562 -0700 PDT m=+0.047850993 container remove c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:21 pr-md-02 systemd[1]: libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 57f8febe-215e-403c-821d-95da6e3c5073 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 2637522f-3bd6-453c-976c-f7da78d12399 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b0cc7927-d150-4870-8ec7-6ef7d847ffd2 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9b053170-587d-42ee-929e-25112b3abba1 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 352cfbfb-4b0a-420c-959b-e42615f9913d does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 144dbb62-1eeb-49ba-b928-eee4a1756251 does not exist May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225286: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225287: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225288: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:29 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225289: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:31 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225290: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:33 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225291: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:35 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225292: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:37 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225293: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:39 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225294: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:41 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225295: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:43 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225296: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:45 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225297: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:46] "GET /metrics HTTP/1.1" 200 73538 "" "Prometheus/2.43.0" May 7 17:07:47 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225298: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:49 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225299: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:51 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225300: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:53 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225301: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225302: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:56] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:07:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225303: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:59 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225304: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225305: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225306: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:04 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225307: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub starts May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub ok May 7 17:08:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:06] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:08:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225308: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 e5615: 46 total, 46 up, 46 in May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:08:07 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', 'cephfs_data', '.mgr', 'cephfs_metadata'] May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:08:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:08:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225310: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:09 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:08:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225311: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225312: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:08:14 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225313: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:16] "GET /metrics HTTP/1.1" 200 73536 "" "Prometheus/2.43.0" May 7 17:08:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225314: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228441. Immutable memtables: 0. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.105603) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12102] Flushing memtable with next log file: 228441 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898105644, "job": 12102, "event": "flush_started", "num_memtables": 1, "num_entries": 2046, "num_deletes": 617, "total_data_size": 1704550, "memory_usage": 1744008, "flush_reason": "Manual Compaction"} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12102] Level-0 flush table #228442: started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898113765, "cf_name": "default", "job": 12102, "event": "table_file_creation", "file_number": 228442, "file_size": 847046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141705570, "largest_seqno": 141707611, "table_properties": {"data_size": 839933, "index_size": 2991, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 32187, "raw_average_key_size": 26, "raw_value_size": 819966, "raw_average_value_size": 670, "num_data_blocks": 115, "num_entries": 1223, "num_filter_entries": 1223, "num_deletions": 617, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126820, "oldest_key_time": 1715126820, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228442, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12102] Flush lasted 8207 microseconds, and 2745 cpu microseconds. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113810) [db/flush_job.cc:967] [default] [JOB 12102] Level-0 flush table #228442: 847046 bytes OK May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113830) [db/memtable_list.cc:519] [default] Level-0 commit table #228442 started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116583) [db/memtable_list.cc:722] [default] Level-0 commit table #228442: memtable #1 done May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116634) EVENT_LOG_v1 {"time_micros": 1715126898116612, "job": 12102, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116650) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12102] Try to delete WAL files size 1693223, prev total WAL file size 1693550, number of live WAL files 2. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228438.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.117035) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740032393030383037' seq:72057594037927935, type:22 .. '6D6772737461740032393031303630' seq:0, type:0; will stop at (end) May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12103] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12102 Base level 0, inputs: [228442(827KB)], [228440(36MB)] May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898117062, "job": 12103, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228442], "files_L6": [228440], "score": -1, "input_data_size": 39098338, "oldest_snapshot_seqno": -1} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12103] Generated table #228443: 25379 keys, 37258556 bytes, temperature: kUnknown May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898320688, "cf_name": "default", "job": 12103, "event": "table_file_creation", "file_number": 228443, "file_size": 37258556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37107602, "index_size": 86411, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702042, "raw_average_key_size": 27, "raw_value_size": 36662378, "raw_average_value_size": 1444, "num_data_blocks": 3038, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228443, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.321010) [db/compaction/compaction_job.cc:1663] [default] [JOB 12103] Compacted 1@0 + 1@6 files to L6 => 37258556 bytes May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323275) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 191.9 rd, 182.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.8, 36.5 +0.0 blob) out(35.5 +0.0 blob), read-write-amplify(90.1) write-amplify(44.0) OK, records in: 26602, records dropped: 1223 output_compression: NoCompression May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323329) EVENT_LOG_v1 {"time_micros": 1715126898323308, "job": 12103, "event": "compaction_finished", "compaction_time_micros": 203764, "compaction_time_cpu_micros": 65018, "output_level": 6, "num_output_files": 1, "total_output_size": 37258556, "num_input_records": 26602, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898323660, "job": 12103, "event": "table_file_deletion", "file_number": 228442} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898326958, "job": 12103, "event": "table_file_deletion", "file_number": 228440} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116998) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326990) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326994) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327002) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327006) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327008) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
I have not updated any packages or anything, I'm not sure how this came about. I did set the following two items earlier today, although I'm not sure if it would make any difference?
ceph config set mds mds_recall_max_caps 3000 ceph config set mds mds_cache_memory_limit 16000000000
I tried to restart the MDS daemons alike so:
ceph orch daemon restart mds.slugfs.pr-md-02.sbblqq
And they seem to start for a moment then stop again.
I then tried to restart the one remaining one that was online, then it seemed to die as well, and now I see this:
# ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
But the ceph-mds on pr-md-01 doesn't exist now.
Anyone know how to recover from this? I'm kind of panicking. :(
Thanks much, erich
OK, we also tried reducing our three MDS daemons to only one: # ceph orch apply mds slugfs --placement="1 pr-md-02.prism" But it still seems stuck: # ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T Logs from pr-md-02.prism below: # tail -f /var/log/messages May 7 20:25:53 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231554: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:25:56] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:25:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231555: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:57 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:25:58 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231556: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231557: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:02 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231558: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:03 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231559: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9abdbcfe-a5ee-45ca-b32b-0a9b75a5b2f8 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 6bbf1d1b-469f-4971-a217-ba32c62c9328 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 411ec1ac-56f3-4b60-b025-eaf2996e2705 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:06] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.080404999 -0700 PDT m=+0.038892155 container create 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.109800855 -0700 PDT m=+0.068287973 container init 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:07 pr-md-02 charming_shaw[2767109]: 167 167 May 7 20:26:07 pr-md-02 systemd[1]: libpod-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116328625 -0700 PDT m=+0.074815739 container start 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116466001 -0700 PDT m=+0.074953116 container attach 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116702031 -0700 PDT m=+0.075189148 container died 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.065184403 -0700 PDT m=+0.023671520 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 podman[2767114]: 2024-05-07 20:26:07.17137253 -0700 PDT m=+0.048040950 container remove 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) May 7 20:26:07 pr-md-02 systemd[1]: libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.294528483 -0700 PDT m=+0.038700338 container create 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.326042029 -0700 PDT m=+0.070213877 container init 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.33345637 -0700 PDT m=+0.077628218 container start 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.333546448 -0700 PDT m=+0.077718304 container attach 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231560: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.280097752 -0700 PDT m=+0.024269612 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> passed data devices: 2 physical, 0 LVM May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> relative data size: 1.0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> All data devices are unavailable May 7 20:26:07 pr-md-02 systemd[1]: libpod-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.965949287 -0700 PDT m=+0.710121150 container died 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True) May 7 20:26:08 pr-md-02 podman[2767179]: 2024-05-07 20:26:08.023360488 -0700 PDT m=+0.050079180 container remove 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-75dbf75fb6c31dcc473ffd4ca180f13e739046bb45c7161fb98d3db9afd17a40-merged.mount: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.57429109 -0700 PDT m=+0.039861869 container create 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.604945119 -0700 PDT m=+0.070515899 container init 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:08 pr-md-02 inspiring_elion[2767333]: 167 167 May 7 20:26:08 pr-md-02 systemd[1]: libpod-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612050835 -0700 PDT m=+0.077621696 container start 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612246287 -0700 PDT m=+0.077817071 container attach 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612503203 -0700 PDT m=+0.078073975 container died 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.license=GPLv2) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.558251332 -0700 PDT m=+0.023822115 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:08 pr-md-02 podman[2767338]: 2024-05-07 20:26:08.670336725 -0700 PDT m=+0.047749623 container remove 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.800520351 -0700 PDT m=+0.040282530 container create 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.834540782 -0700 PDT m=+0.074302964 container init 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.8432442 -0700 PDT m=+0.083006378 container start 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.843360003 -0700 PDT m=+0.083122206 container attach 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.783755475 -0700 PDT m=+0.023517683 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-047f14812498e50e518bc4b80f672140ffa1509b5d508aa134fcb20a609a3189-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "41": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme0n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "41", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "43": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme1n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "43", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ] May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 systemd[1]: libpod-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767359]: 2024-05-07 20:26:09.219197894 -0700 PDT m=+0.458960105 container died 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-f96acd560a7d6f711091ba778252072d08b84d701640411f20ac2c0a30d0d0a0-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767388]: 2024-05-07 20:26:09.276980938 -0700 PDT m=+0.050710570 container remove 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, ceph=True) May 7 20:26:09 pr-md-02 systemd[1]: libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 20:26:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231561: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.236422812 -0700 PDT m=+0.037690490 container create 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.266363875 -0700 PDT m=+0.067631554 container init 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:10 pr-md-02 objective_goldstine[2767540]: 167 167 May 7 20:26:10 pr-md-02 systemd[1]: libpod-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273111289 -0700 PDT m=+0.074378967 container start 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273212161 -0700 PDT m=+0.074479839 container attach 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273506844 -0700 PDT m=+0.074774521 container died 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.222032919 -0700 PDT m=+0.023300596 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:10 pr-md-02 podman[2767545]: 2024-05-07 20:26:10.329094187 -0700 PDT m=+0.049135572 container remove 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 systemd[1]: libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.441995116 -0700 PDT m=+0.040292335 container create c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.475459241 -0700 PDT m=+0.073756409 container init c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482648828 -0700 PDT m=+0.080945996 container start c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482737614 -0700 PDT m=+0.081034798 container attach c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.426366966 -0700 PDT m=+0.024664165 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-30d82c4f42c9c8e9838529715b3cb7fd1874f71c37de5570564e1346385ea590-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231562: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 43, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: }, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 41, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 systemd[1]: libpod-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767566]: 2024-05-07 20:26:11.468853634 -0700 PDT m=+1.067150811 container died c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8b793e8aa6dce35c6bd88479a14748d9d2eeba53693b0716213fb533d5bd9a2b-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767662]: 2024-05-07 20:26:11.526340084 -0700 PDT m=+0.050031409 container remove c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a64027d0-50d9-4df4-84e0-2d6fc7c28d01 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9c614f97-f8ac-4106-bdc4-22a735263ae7 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 087c2c0d-6427-4c79-8604-8e8697a30ad3 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 92b105d6-1f55-4259-8118-6a5ff6cbf800 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 37ac8996-faa0-47bc-8584-d323f77d7b3f does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev f8b7279c-f34c-4451-9400-f78a5674f672 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_03:26:11 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_metadata', '.mgr', 'cephfs_data', 'cephfs_md_and_data'] May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231563: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:13 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231564: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon) e5 handle_command mon_command({"prefix": "status"} v 0) v1 May 7 20:26:15 pr-md-02 ceph-mon[5348]: log_channel(audit) log [DBG] : from='client.? 10.50.1.74:0/2874616819' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch May 7 20:26:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:16] "GET /metrics HTTP/1.1" 200 73366 "" "Prometheus/2.43.0" May 7 20:26:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231565: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:18 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.289994 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231566: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 20:26:21 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231567: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228921. Immutable memtables: 0. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.115346) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12422] Flushing memtable with next log file: 228921 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782115376, "job": 12422, "event": "flush_started", "num_memtables": 1, "num_entries": 3185, "num_deletes": 724, "total_data_size": 2772294, "memory_usage": 2834928, "flush_reason": "Manual Compaction"} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12422] Level-0 flush table #228922: started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782128750, "cf_name": "default", "job": 12422, "event": "table_file_creation", "file_number": 228922, "file_size": 1762231, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142008149, "largest_seqno": 142011329, "table_properties": {"data_size": 1750535, "index_size": 6357, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4293, "raw_key_size": 43713, "raw_average_key_size": 25, "raw_value_size": 1720798, "raw_average_value_size": 1017, "num_data_blocks": 243, "num_entries": 1692, "num_filter_entries": 1692, "num_deletions": 724, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715138637, "oldest_key_time": 1715138637, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228922, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12422] Flush lasted 13451 microseconds, and 5016 cpu microseconds. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128796) [db/flush_job.cc:967] [default] [JOB 12422] Level-0 flush table #228922: 1762231 bytes OK May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128813) [db/memtable_list.cc:519] [default] Level-0 commit table #228922 started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131803) [db/memtable_list.cc:722] [default] Level-0 commit table #228922: memtable #1 done May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131858) EVENT_LOG_v1 {"time_micros": 1715138782131834, "job": 12422, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131894) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12422] Try to delete WAL files size 2755983, prev total WAL file size 2755983, number of live WAL files 2. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228918.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132654) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313539323235' seq:72057594037927935, type:22 .. '7061786F73003130313539343737' seq:0, type:0; will stop at (end) May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12423] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12422 Base level 0, inputs: [228922(1720KB)], [228920(30MB)] May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782132682, "job": 12423, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228922], "files_L6": [228920], "score": -1, "input_data_size": 33301420, "oldest_snapshot_seqno": -1} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12423] Generated table #228923: 24931 keys, 32057163 bytes, temperature: kUnknown May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782250508, "cf_name": "default", "job": 12423, "event": "table_file_creation", "file_number": 228923, "file_size": 32057163, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 31912742, "index_size": 81030, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62341, "raw_key_size": 693222, "raw_average_key_size": 27, "raw_value_size": 31478263, "raw_average_value_size": 1262, "num_data_blocks": 2833, "num_entries": 24931, "num_filter_entries": 24931, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228923, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.250767) [db/compaction/compaction_job.cc:1663] [default] [JOB 12423] Compacted 1@0 + 1@6 files to L6 => 32057163 bytes May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253022) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 282.5 rd, 271.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.7, 30.1 +0.0 blob) out(30.6 +0.0 blob), read-write-amplify(37.1) write-amplify(18.2) OK, records in: 26401, records dropped: 1470 output_compression: NoCompression May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253076) EVENT_LOG_v1 {"time_micros": 1715138782253053, "job": 12423, "event": "compaction_finished", "compaction_time_micros": 117880, "compaction_time_cpu_micros": 56129, "output_level": 6, "num_output_files": 1, "total_output_size": 32057163, "num_input_records": 26401, "num_output_records": 24931, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228922.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782253451, "job": 12423, "event": "table_file_deletion", "file_number": 228922} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228920.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782256177, "job": 12423, "event": "table_file_deletion", "file_number": 228920} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132618) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256209) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256215) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256218) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256221) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:23 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231568: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:23 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata On 5/7/24 6:05 PM, Erich Weiler wrote:
Actually looking again, I don't see core dumps (though I could have sworn I did before). But looking at my MDS servers, they appear to be in a constant cycle of trying to restart, and not making it, then trying again, etc. The logs look like what I sent in my previous email, over and over again.
I tried restarting all three of my ceph-mds daemons at the same time, and it still doesn't seem to help.
I hope I can recover...
-erich
On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,
We just noticed that our cephfs filesystem was unreachable and we saw this:
# ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w)
data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep
io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr
[root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more
And filesystem status is:
# ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there:
# tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225254: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225255: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225256: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:26] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225257: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225258: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225259: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225260: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225261: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:36] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225262: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225263: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225264: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225265: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225266: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:46] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:06:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225267: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225268: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225269: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225270: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225271: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:56] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225272: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225273: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228438. Immutable memtables: 0. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.708981) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12100] Flushing memtable with next log file: 228438 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819709068, "job": 12100, "event": "flush_started", "num_memtables": 1, "num_entries": 3587, "num_deletes": 921, "total_data_size": 2910092, "memory_usage": 2992096, "flush_reason": "Manual Compaction"} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12100] Level-0 flush table #228439: started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819721834, "cf_name": "default", "job": 12100, "event": "table_file_creation", "file_number": 228439, "file_size": 1847607, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141701983, "largest_seqno": 141705565, "table_properties": {"data_size": 1834448, "index_size": 6860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5253, "raw_key_size": 55412, "raw_average_key_size": 26, "raw_value_size": 1798552, "raw_average_value_size": 857, "num_data_blocks": 265, "num_entries": 2097, "num_filter_entries": 2097, "num_deletions": 921, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126677, "oldest_key_time": 1715126677, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228439, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12100] Flush lasted 12882 microseconds, and 5541 cpu microseconds. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721870) [db/flush_job.cc:967] [default] [JOB 12100] Level-0 flush table #228439: 1847607 bytes OK May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721889) [db/memtable_list.cc:519] [default] Level-0 commit table #228439 started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724483) [db/memtable_list.cc:722] [default] Level-0 commit table #228439: memtable #1 done May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724523) EVENT_LOG_v1 {"time_micros": 1715126819724510, "job": 12100, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724561) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12100] Try to delete WAL files size 2890782, prev total WAL file size 2890782, number of live WAL files 2. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725271) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337383930' seq:72057594037927935, type:22 .. '7061786F73003130313338313432' seq:0, type:0; will stop at (end) May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12101] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12100 Base level 0, inputs: [228439(1804KB)], [228437(36MB)] May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819725328, "job": 12101, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228439], "files_L6": [228437], "score": -1, "input_data_size": 39995951, "oldest_snapshot_seqno": -1} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12101] Generated table #228440: 25379 keys, 38251292 bytes, temperature: kUnknown May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819875843, "cf_name": "default", "job": 12101, "event": "table_file_creation", "file_number": 228440, "file_size": 38251292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 38096937, "index_size": 89812, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702456, "raw_average_key_size": 27, "raw_value_size": 37648320, "raw_average_value_size": 1483, "num_data_blocks": 3163, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228440, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.876095) [db/compaction/compaction_job.cc:1663] [default] [JOB 12101] Compacted 1@0 + 1@6 files to L6 => 38251292 bytes May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878629) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 265.6 rd, 254.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.8, 36.4 +0.0 blob) out(36.5 +0.0 blob), read-write-amplify(42.4) write-amplify(20.7) OK, records in: 27250, records dropped: 1871 output_compression: NoCompression May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878669) EVENT_LOG_v1 {"time_micros": 1715126819878658, "job": 12101, "event": "compaction_finished", "compaction_time_micros": 150577, "compaction_time_cpu_micros": 64114, "output_level": 6, "num_output_files": 1, "total_output_size": 38251292, "num_input_records": 27250, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819878937, "job": 12101, "event": "table_file_deletion", "file_number": 228439} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819882652, "job": 12101, "event": "table_file_deletion", "file_number": 228437} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882737) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882741) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882743) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882744) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882746) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:07:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225274: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225275: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225276: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:06] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:07:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225277: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:07:07 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_data', 'cephfs_md_and_data', '.mgr', 'cephfs_metadata'] May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:07:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:07:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225278: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225279: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225280: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:07:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225281: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 e5614: 46 total, 46 up, 46 in May 7 17:07:15 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225283: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d807ba79-465d-4d89-ab03-9bff6275dd10 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9e26e8f6-fba9-425c-923b-ba9da32693a7 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1f435daa-e8f0-4964-b7fb-07ea14b95e9f does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.694558647 -0700 PDT m=+0.039788036 container create 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.722272344 -0700 PDT m=+0.067501679 container init 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:07:17 pr-md-02 stoic_mayer[2565752]: 167 167 May 7 17:07:17 pr-md-02 systemd[1]: libpod-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728419093 -0700 PDT m=+0.073648421 container start 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728522035 -0700 PDT m=+0.073751360 container attach 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728769341 -0700 PDT m=+0.073998662 container died 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.678321769 -0700 PDT m=+0.023551090 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:17 pr-md-02 podman[2565757]: 2024-05-07 17:07:17.78259101 -0700 PDT m=+0.046708660 container remove 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 systemd[1]: libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.90851263 -0700 PDT m=+0.039669496 container create 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.936966849 -0700 PDT m=+0.068123675 container init 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943702066 -0700 PDT m=+0.074858891 container start 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943794994 -0700 PDT m=+0.074951827 container attach 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.892929482 -0700 PDT m=+0.024086307 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> passed data devices: 2 physical, 0 LVM May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> relative data size: 1.0 May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> All data devices are unavailable May 7 17:07:18 pr-md-02 systemd[1]: libpod-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 podman[2565778]: 2024-05-07 17:07:18.556739188 -0700 PDT m=+0.687896012 container died 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:18 pr-md-02 podman[2565822]: 2024-05-07 17:07:18.614711286 -0700 PDT m=+0.047414226 container remove 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:18 pr-md-02 systemd[1]: libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-2f63fd9ffd2763a7ee3f75befaf99b27a75325cd025cc2e74e72d7bd150afc42-merged.mount: Deactivated successfully. May 7 17:07:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225284: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.171860525 -0700 PDT m=+0.039261474 container create eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.199519439 -0700 PDT m=+0.066920407 container init eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:19 pr-md-02 xenodochial_wu[2565974]: 167 167 May 7 17:07:19 pr-md-02 systemd[1]: libpod-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.205924433 -0700 PDT m=+0.073325392 container start eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.2060646 -0700 PDT m=+0.073465559 container attach eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.206298065 -0700 PDT m=+0.073699031 container died eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.158156321 -0700 PDT m=+0.025557273 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 podman[2565979]: 2024-05-07 17:07:19.25994464 -0700 PDT m=+0.046894141 container remove eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.388083595 -0700 PDT m=+0.043236647 container create a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.417053374 -0700 PDT m=+0.072206356 container init a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423715752 -0700 PDT m=+0.078868733 container start a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423810643 -0700 PDT m=+0.078963634 container attach a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.371099332 -0700 PDT m=+0.026252317 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-91f5899d00d5794e67bbdd45fa72d22c74b0b977efec0238aa842ac272b635ce-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "41": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme0n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "41", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "43": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme1n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "43", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ] May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 systemd[1]: libpod-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.801695942 -0700 PDT m=+0.456848932 container died a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-68bc380ea9ec1cff282d08f862c7b661ba0571a20622225a5a3994e5fc88985d-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566028]: 2024-05-07 17:07:19.854983202 -0700 PDT m=+0.045986655 container remove a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.420915201 -0700 PDT m=+0.041065865 container create f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.45143613 -0700 PDT m=+0.071586769 container init f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:07:20 pr-md-02 practical_wu[2566180]: 167 167 May 7 17:07:20 pr-md-02 systemd[1]: libpod-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457513493 -0700 PDT m=+0.077664122 container start f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457638676 -0700 PDT m=+0.077789308 container attach f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457913249 -0700 PDT m=+0.078063874 container died f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.405495878 -0700 PDT m=+0.025646501 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 podman[2566185]: 2024-05-07 17:07:20.515553442 -0700 PDT m=+0.045837844 container remove f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) May 7 17:07:20 pr-md-02 systemd[1]: libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.631476902 -0700 PDT m=+0.043759539 container create c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.660235834 -0700 PDT m=+0.072518401 container init c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.666915701 -0700 PDT m=+0.079198260 container start c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.667002215 -0700 PDT m=+0.079284787 container attach c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) May 7 17:07:20 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb2679257a24186947b435237af458658d281389b531640688ba39cfa7cac79-merged.mount: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.6134279 -0700 PDT m=+0.025710470 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225285: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 43, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: }, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 41, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 systemd[1]: libpod-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566207]: 2024-05-07 17:07:21.655826716 -0700 PDT m=+1.068109291 container died c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:21 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-e573f9cf164334d9b913d82ba760fa7f00e98646ea067755c99a3842969d30d8-merged.mount: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566303]: 2024-05-07 17:07:21.711419562 -0700 PDT m=+0.047850993 container remove c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:21 pr-md-02 systemd[1]: libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 57f8febe-215e-403c-821d-95da6e3c5073 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 2637522f-3bd6-453c-976c-f7da78d12399 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b0cc7927-d150-4870-8ec7-6ef7d847ffd2 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9b053170-587d-42ee-929e-25112b3abba1 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 352cfbfb-4b0a-420c-959b-e42615f9913d does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 144dbb62-1eeb-49ba-b928-eee4a1756251 does not exist May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225286: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225287: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225288: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:29 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225289: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:31 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225290: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:33 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225291: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:35 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225292: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:37 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225293: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:39 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225294: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:41 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225295: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:43 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225296: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:45 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225297: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:46] "GET /metrics HTTP/1.1" 200 73538 "" "Prometheus/2.43.0" May 7 17:07:47 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225298: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:49 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225299: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:51 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225300: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:53 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225301: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225302: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:56] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:07:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225303: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:59 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225304: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225305: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225306: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:04 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225307: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub starts May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub ok May 7 17:08:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:06] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:08:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225308: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 e5615: 46 total, 46 up, 46 in May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:08:07 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', 'cephfs_data', '.mgr', 'cephfs_metadata'] May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:08:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:08:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225310: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:09 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:08:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225311: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225312: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:08:14 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225313: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:16] "GET /metrics HTTP/1.1" 200 73536 "" "Prometheus/2.43.0" May 7 17:08:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225314: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228441. Immutable memtables: 0. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.105603) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12102] Flushing memtable with next log file: 228441 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898105644, "job": 12102, "event": "flush_started", "num_memtables": 1, "num_entries": 2046, "num_deletes": 617, "total_data_size": 1704550, "memory_usage": 1744008, "flush_reason": "Manual Compaction"} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12102] Level-0 flush table #228442: started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898113765, "cf_name": "default", "job": 12102, "event": "table_file_creation", "file_number": 228442, "file_size": 847046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141705570, "largest_seqno": 141707611, "table_properties": {"data_size": 839933, "index_size": 2991, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 32187, "raw_average_key_size": 26, "raw_value_size": 819966, "raw_average_value_size": 670, "num_data_blocks": 115, "num_entries": 1223, "num_filter_entries": 1223, "num_deletions": 617, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126820, "oldest_key_time": 1715126820, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228442, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12102] Flush lasted 8207 microseconds, and 2745 cpu microseconds. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113810) [db/flush_job.cc:967] [default] [JOB 12102] Level-0 flush table #228442: 847046 bytes OK May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113830) [db/memtable_list.cc:519] [default] Level-0 commit table #228442 started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116583) [db/memtable_list.cc:722] [default] Level-0 commit table #228442: memtable #1 done May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116634) EVENT_LOG_v1 {"time_micros": 1715126898116612, "job": 12102, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116650) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12102] Try to delete WAL files size 1693223, prev total WAL file size 1693550, number of live WAL files 2. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228438.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.117035) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740032393030383037' seq:72057594037927935, type:22 .. '6D6772737461740032393031303630' seq:0, type:0; will stop at (end) May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12103] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12102 Base level 0, inputs: [228442(827KB)], [228440(36MB)] May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898117062, "job": 12103, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228442], "files_L6": [228440], "score": -1, "input_data_size": 39098338, "oldest_snapshot_seqno": -1} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12103] Generated table #228443: 25379 keys, 37258556 bytes, temperature: kUnknown May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898320688, "cf_name": "default", "job": 12103, "event": "table_file_creation", "file_number": 228443, "file_size": 37258556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37107602, "index_size": 86411, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702042, "raw_average_key_size": 27, "raw_value_size": 36662378, "raw_average_value_size": 1444, "num_data_blocks": 3038, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228443, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.321010) [db/compaction/compaction_job.cc:1663] [default] [JOB 12103] Compacted 1@0 + 1@6 files to L6 => 37258556 bytes May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323275) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 191.9 rd, 182.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.8, 36.5 +0.0 blob) out(35.5 +0.0 blob), read-write-amplify(90.1) write-amplify(44.0) OK, records in: 26602, records dropped: 1223 output_compression: NoCompression May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323329) EVENT_LOG_v1 {"time_micros": 1715126898323308, "job": 12103, "event": "compaction_finished", "compaction_time_micros": 203764, "compaction_time_cpu_micros": 65018, "output_level": 6, "num_output_files": 1, "total_output_size": 37258556, "num_input_records": 26602, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898323660, "job": 12103, "event": "table_file_deletion", "file_number": 228442} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898326958, "job": 12103, "event": "table_file_deletion", "file_number": 228440} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116998) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326990) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326994) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327002) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327006) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327008) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
I have not updated any packages or anything, I'm not sure how this came about. I did set the following two items earlier today, although I'm not sure if it would make any difference?
ceph config set mds mds_recall_max_caps 3000 ceph config set mds mds_cache_memory_limit 16000000000
I tried to restart the MDS daemons alike so:
ceph orch daemon restart mds.slugfs.pr-md-02.sbblqq
And they seem to start for a moment then stop again.
I then tried to restart the one remaining one that was online, then it seemed to die as well, and now I see this:
# ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
But the ceph-mds on pr-md-01 doesn't exist now.
Anyone know how to recover from this? I'm kind of panicking. :(
Thanks much, erich
Hi Erich, I don't see anything useful in the syslog output. Can you instead paste only the mds log? On the mds host you can run cephadm logs --name mds.slugfs.pr-md-02.sbblqq or journalctl -u ceph-{FSID}@mds.slugfs.pr-md-02.sbblqq Zitat von Erich Weiler <weiler@soe.ucsc.edu>:
OK, we also tried reducing our three MDS daemons to only one:
# ceph orch apply mds slugfs --placement="1 pr-md-02.prism"
But it still seems stuck:
# ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T
Logs from pr-md-02.prism below:
# tail -f /var/log/messages May 7 20:25:53 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231554: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:25:56] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:25:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231555: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:57 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:25:58 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231556: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231557: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:02 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231558: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:03 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231559: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9abdbcfe-a5ee-45ca-b32b-0a9b75a5b2f8 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 6bbf1d1b-469f-4971-a217-ba32c62c9328 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 411ec1ac-56f3-4b60-b025-eaf2996e2705 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:06] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.080404999 -0700 PDT m=+0.038892155 container create 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.109800855 -0700 PDT m=+0.068287973 container init 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:07 pr-md-02 charming_shaw[2767109]: 167 167 May 7 20:26:07 pr-md-02 systemd[1]: libpod-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116328625 -0700 PDT m=+0.074815739 container start 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116466001 -0700 PDT m=+0.074953116 container attach 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116702031 -0700 PDT m=+0.075189148 container died 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.065184403 -0700 PDT m=+0.023671520 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 podman[2767114]: 2024-05-07 20:26:07.17137253 -0700 PDT m=+0.048040950 container remove 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) May 7 20:26:07 pr-md-02 systemd[1]: libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.294528483 -0700 PDT m=+0.038700338 container create 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.326042029 -0700 PDT m=+0.070213877 container init 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.33345637 -0700 PDT m=+0.077628218 container start 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.333546448 -0700 PDT m=+0.077718304 container attach 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231560: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.280097752 -0700 PDT m=+0.024269612 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> passed data devices: 2 physical, 0 LVM May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> relative data size: 1.0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> All data devices are unavailable May 7 20:26:07 pr-md-02 systemd[1]: libpod-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.965949287 -0700 PDT m=+0.710121150 container died 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True) May 7 20:26:08 pr-md-02 podman[2767179]: 2024-05-07 20:26:08.023360488 -0700 PDT m=+0.050079180 container remove 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-75dbf75fb6c31dcc473ffd4ca180f13e739046bb45c7161fb98d3db9afd17a40-merged.mount: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.57429109 -0700 PDT m=+0.039861869 container create 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.604945119 -0700 PDT m=+0.070515899 container init 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:08 pr-md-02 inspiring_elion[2767333]: 167 167 May 7 20:26:08 pr-md-02 systemd[1]: libpod-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612050835 -0700 PDT m=+0.077621696 container start 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612246287 -0700 PDT m=+0.077817071 container attach 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612503203 -0700 PDT m=+0.078073975 container died 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.license=GPLv2) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.558251332 -0700 PDT m=+0.023822115 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:08 pr-md-02 podman[2767338]: 2024-05-07 20:26:08.670336725 -0700 PDT m=+0.047749623 container remove 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.800520351 -0700 PDT m=+0.040282530 container create 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.834540782 -0700 PDT m=+0.074302964 container init 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.8432442 -0700 PDT m=+0.083006378 container start 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.843360003 -0700 PDT m=+0.083122206 container attach 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.783755475 -0700 PDT m=+0.023517683 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-047f14812498e50e518bc4b80f672140ffa1509b5d508aa134fcb20a609a3189-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "41": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme0n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "41", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "43": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme1n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "43", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ] May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 systemd[1]: libpod-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767359]: 2024-05-07 20:26:09.219197894 -0700 PDT m=+0.458960105 container died 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-f96acd560a7d6f711091ba778252072d08b84d701640411f20ac2c0a30d0d0a0-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767388]: 2024-05-07 20:26:09.276980938 -0700 PDT m=+0.050710570 container remove 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, ceph=True) May 7 20:26:09 pr-md-02 systemd[1]: libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 20:26:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231561: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.236422812 -0700 PDT m=+0.037690490 container create 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.266363875 -0700 PDT m=+0.067631554 container init 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:10 pr-md-02 objective_goldstine[2767540]: 167 167 May 7 20:26:10 pr-md-02 systemd[1]: libpod-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273111289 -0700 PDT m=+0.074378967 container start 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273212161 -0700 PDT m=+0.074479839 container attach 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273506844 -0700 PDT m=+0.074774521 container died 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.222032919 -0700 PDT m=+0.023300596 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:10 pr-md-02 podman[2767545]: 2024-05-07 20:26:10.329094187 -0700 PDT m=+0.049135572 container remove 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 systemd[1]: libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.441995116 -0700 PDT m=+0.040292335 container create c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.475459241 -0700 PDT m=+0.073756409 container init c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482648828 -0700 PDT m=+0.080945996 container start c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482737614 -0700 PDT m=+0.081034798 container attach c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.426366966 -0700 PDT m=+0.024664165 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-30d82c4f42c9c8e9838529715b3cb7fd1874f71c37de5570564e1346385ea590-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231562: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 43, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: }, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 41, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 systemd[1]: libpod-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767566]: 2024-05-07 20:26:11.468853634 -0700 PDT m=+1.067150811 container died c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8b793e8aa6dce35c6bd88479a14748d9d2eeba53693b0716213fb533d5bd9a2b-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767662]: 2024-05-07 20:26:11.526340084 -0700 PDT m=+0.050031409 container remove c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a64027d0-50d9-4df4-84e0-2d6fc7c28d01 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9c614f97-f8ac-4106-bdc4-22a735263ae7 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 087c2c0d-6427-4c79-8604-8e8697a30ad3 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 92b105d6-1f55-4259-8118-6a5ff6cbf800 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 37ac8996-faa0-47bc-8584-d323f77d7b3f does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev f8b7279c-f34c-4451-9400-f78a5674f672 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_03:26:11 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_metadata', '.mgr', 'cephfs_data', 'cephfs_md_and_data'] May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231563: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:13 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231564: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon) e5 handle_command mon_command({"prefix": "status"} v 0) v1 May 7 20:26:15 pr-md-02 ceph-mon[5348]: log_channel(audit) log [DBG] : from='client.? 10.50.1.74:0/2874616819' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch May 7 20:26:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:16] "GET /metrics HTTP/1.1" 200 73366 "" "Prometheus/2.43.0" May 7 20:26:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231565: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:18 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.289994 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231566: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 20:26:21 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231567: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228921. Immutable memtables: 0. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.115346) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12422] Flushing memtable with next log file: 228921 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782115376, "job": 12422, "event": "flush_started", "num_memtables": 1, "num_entries": 3185, "num_deletes": 724, "total_data_size": 2772294, "memory_usage": 2834928, "flush_reason": "Manual Compaction"} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12422] Level-0 flush table #228922: started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782128750, "cf_name": "default", "job": 12422, "event": "table_file_creation", "file_number": 228922, "file_size": 1762231, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142008149, "largest_seqno": 142011329, "table_properties": {"data_size": 1750535, "index_size": 6357, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4293, "raw_key_size": 43713, "raw_average_key_size": 25, "raw_value_size": 1720798, "raw_average_value_size": 1017, "num_data_blocks": 243, "num_entries": 1692, "num_filter_entries": 1692, "num_deletions": 724, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715138637, "oldest_key_time": 1715138637, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228922, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12422] Flush lasted 13451 microseconds, and 5016 cpu microseconds. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128796) [db/flush_job.cc:967] [default] [JOB 12422] Level-0 flush table #228922: 1762231 bytes OK May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128813) [db/memtable_list.cc:519] [default] Level-0 commit table #228922 started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131803) [db/memtable_list.cc:722] [default] Level-0 commit table #228922: memtable #1 done May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131858) EVENT_LOG_v1 {"time_micros": 1715138782131834, "job": 12422, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131894) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12422] Try to delete WAL files size 2755983, prev total WAL file size 2755983, number of live WAL files 2. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228918.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132654) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313539323235' seq:72057594037927935, type:22 .. '7061786F73003130313539343737' seq:0, type:0; will stop at (end) May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12423] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12422 Base level 0, inputs: [228922(1720KB)], [228920(30MB)] May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782132682, "job": 12423, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228922], "files_L6": [228920], "score": -1, "input_data_size": 33301420, "oldest_snapshot_seqno": -1} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12423] Generated table #228923: 24931 keys, 32057163 bytes, temperature: kUnknown May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782250508, "cf_name": "default", "job": 12423, "event": "table_file_creation", "file_number": 228923, "file_size": 32057163, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 31912742, "index_size": 81030, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62341, "raw_key_size": 693222, "raw_average_key_size": 27, "raw_value_size": 31478263, "raw_average_value_size": 1262, "num_data_blocks": 2833, "num_entries": 24931, "num_filter_entries": 24931, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228923, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.250767) [db/compaction/compaction_job.cc:1663] [default] [JOB 12423] Compacted 1@0 + 1@6 files to L6 => 32057163 bytes May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253022) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 282.5 rd, 271.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.7, 30.1 +0.0 blob) out(30.6 +0.0 blob), read-write-amplify(37.1) write-amplify(18.2) OK, records in: 26401, records dropped: 1470 output_compression: NoCompression May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253076) EVENT_LOG_v1 {"time_micros": 1715138782253053, "job": 12423, "event": "compaction_finished", "compaction_time_micros": 117880, "compaction_time_cpu_micros": 56129, "output_level": 6, "num_output_files": 1, "total_output_size": 32057163, "num_input_records": 26401, "num_output_records": 24931, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228922.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782253451, "job": 12423, "event": "table_file_deletion", "file_number": 228922} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228920.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782256177, "job": 12423, "event": "table_file_deletion", "file_number": 228920} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132618) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256209) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256215) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256218) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256221) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:23 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231568: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:23 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata
On 5/7/24 6:05 PM, Erich Weiler wrote:
Actually looking again, I don't see core dumps (though I could have sworn I did before). But looking at my MDS servers, they appear to be in a constant cycle of trying to restart, and not making it, then trying again, etc. The logs look like what I sent in my previous email, over and over again.
I tried restarting all three of my ceph-mds daemons at the same time, and it still doesn't seem to help.
I hope I can recover...
-erich
On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,
We just noticed that our cephfs filesystem was unreachable and we saw this:
# ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w)
data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep
io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr
[root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more
And filesystem status is:
# ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there:
# tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing [Text gekürzt]
My supervisor Jorge replied to Eugen but I'm copying the list with the log file, attached. -erich On 5/8/24 12:24 AM, Eugen Block wrote:
Hi Erich, I don't see anything useful in the syslog output. Can you instead paste only the mds log? On the mds host you can run
cephadm logs --name mds.slugfs.pr-md-02.sbblqq
or
journalctl -u ceph-{FSID}@mds.slugfs.pr-md-02.sbblqq
Zitat von Erich Weiler <weiler@soe.ucsc.edu>:
OK, we also tried reducing our three MDS daemons to only one:
# ceph orch apply mds slugfs --placement="1 pr-md-02.prism"
But it still seems stuck:
# ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T
Logs from pr-md-02.prism below:
# tail -f /var/log/messages May 7 20:25:53 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231554: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:25:56] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:25:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231555: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:57 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:25:58 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231556: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231557: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:02 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231558: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:03 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231559: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9abdbcfe-a5ee-45ca-b32b-0a9b75a5b2f8 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 6bbf1d1b-469f-4971-a217-ba32c62c9328 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 411ec1ac-56f3-4b60-b025-eaf2996e2705 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:06] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.080404999 -0700 PDT m=+0.038892155 container create 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.109800855 -0700 PDT m=+0.068287973 container init 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:07 pr-md-02 charming_shaw[2767109]: 167 167 May 7 20:26:07 pr-md-02 systemd[1]: libpod-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116328625 -0700 PDT m=+0.074815739 container start 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116466001 -0700 PDT m=+0.074953116 container attach 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116702031 -0700 PDT m=+0.075189148 container died 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.065184403 -0700 PDT m=+0.023671520 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 podman[2767114]: 2024-05-07 20:26:07.17137253 -0700 PDT m=+0.048040950 container remove 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) May 7 20:26:07 pr-md-02 systemd[1]: libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.294528483 -0700 PDT m=+0.038700338 container create 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.326042029 -0700 PDT m=+0.070213877 container init 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.33345637 -0700 PDT m=+0.077628218 container start 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.333546448 -0700 PDT m=+0.077718304 container attach 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231560: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.280097752 -0700 PDT m=+0.024269612 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> passed data devices: 2 physical, 0 LVM May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> relative data size: 1.0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> All data devices are unavailable May 7 20:26:07 pr-md-02 systemd[1]: libpod-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.965949287 -0700 PDT m=+0.710121150 container died 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True) May 7 20:26:08 pr-md-02 podman[2767179]: 2024-05-07 20:26:08.023360488 -0700 PDT m=+0.050079180 container remove 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-75dbf75fb6c31dcc473ffd4ca180f13e739046bb45c7161fb98d3db9afd17a40-merged.mount: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.57429109 -0700 PDT m=+0.039861869 container create 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.604945119 -0700 PDT m=+0.070515899 container init 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:08 pr-md-02 inspiring_elion[2767333]: 167 167 May 7 20:26:08 pr-md-02 systemd[1]: libpod-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612050835 -0700 PDT m=+0.077621696 container start 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612246287 -0700 PDT m=+0.077817071 container attach 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612503203 -0700 PDT m=+0.078073975 container died 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.license=GPLv2) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.558251332 -0700 PDT m=+0.023822115 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:08 pr-md-02 podman[2767338]: 2024-05-07 20:26:08.670336725 -0700 PDT m=+0.047749623 container remove 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.800520351 -0700 PDT m=+0.040282530 container create 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.834540782 -0700 PDT m=+0.074302964 container init 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.8432442 -0700 PDT m=+0.083006378 container start 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.843360003 -0700 PDT m=+0.083122206 container attach 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.783755475 -0700 PDT m=+0.023517683 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-047f14812498e50e518bc4b80f672140ffa1509b5d508aa134fcb20a609a3189-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "41": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme0n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "41", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "43": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme1n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "43", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ] May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 systemd[1]: libpod-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767359]: 2024-05-07 20:26:09.219197894 -0700 PDT m=+0.458960105 container died 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-f96acd560a7d6f711091ba778252072d08b84d701640411f20ac2c0a30d0d0a0-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767388]: 2024-05-07 20:26:09.276980938 -0700 PDT m=+0.050710570 container remove 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, ceph=True) May 7 20:26:09 pr-md-02 systemd[1]: libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 20:26:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231561: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.236422812 -0700 PDT m=+0.037690490 container create 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.266363875 -0700 PDT m=+0.067631554 container init 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:10 pr-md-02 objective_goldstine[2767540]: 167 167 May 7 20:26:10 pr-md-02 systemd[1]: libpod-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273111289 -0700 PDT m=+0.074378967 container start 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273212161 -0700 PDT m=+0.074479839 container attach 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273506844 -0700 PDT m=+0.074774521 container died 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.222032919 -0700 PDT m=+0.023300596 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:10 pr-md-02 podman[2767545]: 2024-05-07 20:26:10.329094187 -0700 PDT m=+0.049135572 container remove 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 systemd[1]: libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.441995116 -0700 PDT m=+0.040292335 container create c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.475459241 -0700 PDT m=+0.073756409 container init c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482648828 -0700 PDT m=+0.080945996 container start c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482737614 -0700 PDT m=+0.081034798 container attach c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.426366966 -0700 PDT m=+0.024664165 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-30d82c4f42c9c8e9838529715b3cb7fd1874f71c37de5570564e1346385ea590-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231562: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 43, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: }, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 41, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 systemd[1]: libpod-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767566]: 2024-05-07 20:26:11.468853634 -0700 PDT m=+1.067150811 container died c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8b793e8aa6dce35c6bd88479a14748d9d2eeba53693b0716213fb533d5bd9a2b-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767662]: 2024-05-07 20:26:11.526340084 -0700 PDT m=+0.050031409 container remove c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a64027d0-50d9-4df4-84e0-2d6fc7c28d01 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9c614f97-f8ac-4106-bdc4-22a735263ae7 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 087c2c0d-6427-4c79-8604-8e8697a30ad3 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 92b105d6-1f55-4259-8118-6a5ff6cbf800 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 37ac8996-faa0-47bc-8584-d323f77d7b3f does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev f8b7279c-f34c-4451-9400-f78a5674f672 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_03:26:11 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_metadata', '.mgr', 'cephfs_data', 'cephfs_md_and_data'] May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231563: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:13 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231564: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon) e5 handle_command mon_command({"prefix": "status"} v 0) v1 May 7 20:26:15 pr-md-02 ceph-mon[5348]: log_channel(audit) log [DBG] : from='client.? 10.50.1.74:0/2874616819' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch May 7 20:26:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:16] "GET /metrics HTTP/1.1" 200 73366 "" "Prometheus/2.43.0" May 7 20:26:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231565: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:18 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.289994 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231566: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 20:26:21 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231567: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228921. Immutable memtables: 0. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.115346) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12422] Flushing memtable with next log file: 228921 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782115376, "job": 12422, "event": "flush_started", "num_memtables": 1, "num_entries": 3185, "num_deletes": 724, "total_data_size": 2772294, "memory_usage": 2834928, "flush_reason": "Manual Compaction"} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12422] Level-0 flush table #228922: started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782128750, "cf_name": "default", "job": 12422, "event": "table_file_creation", "file_number": 228922, "file_size": 1762231, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142008149, "largest_seqno": 142011329, "table_properties": {"data_size": 1750535, "index_size": 6357, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4293, "raw_key_size": 43713, "raw_average_key_size": 25, "raw_value_size": 1720798, "raw_average_value_size": 1017, "num_data_blocks": 243, "num_entries": 1692, "num_filter_entries": 1692, "num_deletions": 724, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715138637, "oldest_key_time": 1715138637, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228922, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12422] Flush lasted 13451 microseconds, and 5016 cpu microseconds. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128796) [db/flush_job.cc:967] [default] [JOB 12422] Level-0 flush table #228922: 1762231 bytes OK May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128813) [db/memtable_list.cc:519] [default] Level-0 commit table #228922 started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131803) [db/memtable_list.cc:722] [default] Level-0 commit table #228922: memtable #1 done May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131858) EVENT_LOG_v1 {"time_micros": 1715138782131834, "job": 12422, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131894) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12422] Try to delete WAL files size 2755983, prev total WAL file size 2755983, number of live WAL files 2. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228918.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132654) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313539323235' seq:72057594037927935, type:22 .. '7061786F73003130313539343737' seq:0, type:0; will stop at (end) May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12423] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12422 Base level 0, inputs: [228922(1720KB)], [228920(30MB)] May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782132682, "job": 12423, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228922], "files_L6": [228920], "score": -1, "input_data_size": 33301420, "oldest_snapshot_seqno": -1} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12423] Generated table #228923: 24931 keys, 32057163 bytes, temperature: kUnknown May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782250508, "cf_name": "default", "job": 12423, "event": "table_file_creation", "file_number": 228923, "file_size": 32057163, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 31912742, "index_size": 81030, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62341, "raw_key_size": 693222, "raw_average_key_size": 27, "raw_value_size": 31478263, "raw_average_value_size": 1262, "num_data_blocks": 2833, "num_entries": 24931, "num_filter_entries": 24931, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228923, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.250767) [db/compaction/compaction_job.cc:1663] [default] [JOB 12423] Compacted 1@0 + 1@6 files to L6 => 32057163 bytes May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253022) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 282.5 rd, 271.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.7, 30.1 +0.0 blob) out(30.6 +0.0 blob), read-write-amplify(37.1) write-amplify(18.2) OK, records in: 26401, records dropped: 1470 output_compression: NoCompression May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253076) EVENT_LOG_v1 {"time_micros": 1715138782253053, "job": 12423, "event": "compaction_finished", "compaction_time_micros": 117880, "compaction_time_cpu_micros": 56129, "output_level": 6, "num_output_files": 1, "total_output_size": 32057163, "num_input_records": 26401, "num_output_records": 24931, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228922.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782253451, "job": 12423, "event": "table_file_deletion", "file_number": 228922} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228920.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782256177, "job": 12423, "event": "table_file_deletion", "file_number": 228920} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132618) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256209) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256215) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256218) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256221) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:23 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231568: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:23 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata
On 5/7/24 6:05 PM, Erich Weiler wrote:
Actually looking again, I don't see core dumps (though I could have sworn I did before). But looking at my MDS servers, they appear to be in a constant cycle of trying to restart, and not making it, then trying again, etc. The logs look like what I sent in my previous email, over and over again.
I tried restarting all three of my ceph-mds daemons at the same time, and it still doesn't seem to help.
I hope I can recover...
-erich
On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,
We just noticed that our cephfs filesystem was unreachable and we saw this:
# ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w)
data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep
io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr
[root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more
And filesystem status is:
# ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there:
# tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing [Text gekürzt]
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The stack trace looks exactly like the one in this thread [1]. But I'm not sure how to apply the workaround, maybe stop all client IO and see if the MDS recovers? And then mount the clients with the option nowsync? Hopefully Xiubo (or someone else) will chime in here. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/OEU6PWVR5CQS... Zitat von Erich Weiler <weiler@soe.ucsc.edu>:
My supervisor Jorge replied to Eugen but I'm copying the list with the log file, attached.
-erich
On 5/8/24 12:24 AM, Eugen Block wrote:
Hi Erich, I don't see anything useful in the syslog output. Can you instead paste only the mds log? On the mds host you can run
cephadm logs --name mds.slugfs.pr-md-02.sbblqq
or
journalctl -u ceph-{FSID}@mds.slugfs.pr-md-02.sbblqq
Zitat von Erich Weiler <weiler@soe.ucsc.edu>:
OK, we also tried reducing our three MDS daemons to only one:
# ceph orch apply mds slugfs --placement="1 pr-md-02.prism"
But it still seems stuck:
# ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T
Logs from pr-md-02.prism below:
# tail -f /var/log/messages May 7 20:25:53 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 20:25:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231554: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:25:56] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:25:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231555: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:25:57 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:25:58 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:25:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231556: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231557: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:02 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231558: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:03 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231559: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9abdbcfe-a5ee-45ca-b32b-0a9b75a5b2f8 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 6bbf1d1b-469f-4971-a217-ba32c62c9328 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 411ec1ac-56f3-4b60-b025-eaf2996e2705 does not exist May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:06] "GET /metrics HTTP/1.1" 200 73369 "" "Prometheus/2.43.0" May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.080404999 -0700 PDT m=+0.038892155 container create 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.109800855 -0700 PDT m=+0.068287973 container init 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:07 pr-md-02 charming_shaw[2767109]: 167 167 May 7 20:26:07 pr-md-02 systemd[1]: libpod-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116328625 -0700 PDT m=+0.074815739 container start 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116466001 -0700 PDT m=+0.074953116 container attach 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.116702031 -0700 PDT m=+0.075189148 container died 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) May 7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07 20:26:07.065184403 -0700 PDT m=+0.023671520 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 podman[2767114]: 2024-05-07 20:26:07.17137253 -0700 PDT m=+0.048040950 container remove 4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) May 7 20:26:07 pr-md-02 systemd[1]: libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.294528483 -0700 PDT m=+0.038700338 container create 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1) May 7 20:26:07 pr-md-02 systemd[1]: Started libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope. May 7 20:26:07 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.326042029 -0700 PDT m=+0.070213877 container init 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.33345637 -0700 PDT m=+0.077628218 container start 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.333546448 -0700 PDT m=+0.077718304 container attach 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 20:26:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231560: 1313 pgs: 5 active+clean+scrubbing+deep, 6 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.280097752 -0700 PDT m=+0.024269612 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:07 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> passed data devices: 2 physical, 0 LVM May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> relative data size: 1.0 May 7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> All data devices are unavailable May 7 20:26:07 pr-md-02 systemd[1]: libpod-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07 20:26:07.965949287 -0700 PDT m=+0.710121150 container died 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True) May 7 20:26:08 pr-md-02 podman[2767179]: 2024-05-07 20:26:08.023360488 -0700 PDT m=+0.050079180 container remove 38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-75dbf75fb6c31dcc473ffd4ca180f13e739046bb45c7161fb98d3db9afd17a40-merged.mount: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.57429109 -0700 PDT m=+0.039861869 container create 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.604945119 -0700 PDT m=+0.070515899 container init 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:08 pr-md-02 inspiring_elion[2767333]: 167 167 May 7 20:26:08 pr-md-02 systemd[1]: libpod-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612050835 -0700 PDT m=+0.077621696 container start 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612246287 -0700 PDT m=+0.077817071 container attach 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.612503203 -0700 PDT m=+0.078073975 container died 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.license=GPLv2) May 7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07 20:26:08.558251332 -0700 PDT m=+0.023822115 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:08 pr-md-02 podman[2767338]: 2024-05-07 20:26:08.670336725 -0700 PDT m=+0.047749623 container remove 417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 20:26:08 pr-md-02 systemd[1]: libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated successfully. May 7 20:26:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.800520351 -0700 PDT m=+0.040282530 container create 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:08 pr-md-02 systemd[1]: Started libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope. May 7 20:26:08 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.834540782 -0700 PDT m=+0.074302964 container init 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.8432442 -0700 PDT m=+0.083006378 container start 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.843360003 -0700 PDT m=+0.083122206 container attach 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07 20:26:08.783755475 -0700 PDT m=+0.023517683 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-047f14812498e50e518bc4b80f672140ffa1509b5d508aa134fcb20a609a3189-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "41": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme0n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "41", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "43": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "devices": [ May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme1n1" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ], May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_size": "1920378863616", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "tags": { May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.cluster_name": "ceph", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.crush_device_class": "", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "43", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: }, May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "type": "block", May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: ] May 7 20:26:09 pr-md-02 trusting_noyce[2767376]: } May 7 20:26:09 pr-md-02 systemd[1]: libpod-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767359]: 2024-05-07 20:26:09.219197894 -0700 PDT m=+0.458960105 container died 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:09 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-f96acd560a7d6f711091ba778252072d08b84d701640411f20ac2c0a30d0d0a0-merged.mount: Deactivated successfully. May 7 20:26:09 pr-md-02 podman[2767388]: 2024-05-07 20:26:09.276980938 -0700 PDT m=+0.050710570 container remove 386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, ceph=True) May 7 20:26:09 pr-md-02 systemd[1]: libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated successfully. May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 20:26:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231561: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.236422812 -0700 PDT m=+0.037690490 container create 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.266363875 -0700 PDT m=+0.067631554 container init 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 20:26:10 pr-md-02 objective_goldstine[2767540]: 167 167 May 7 20:26:10 pr-md-02 systemd[1]: libpod-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273111289 -0700 PDT m=+0.074378967 container start 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273212161 -0700 PDT m=+0.074479839 container attach 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.273506844 -0700 PDT m=+0.074774521 container died 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD) May 7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07 20:26:10.222032919 -0700 PDT m=+0.023300596 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:10 pr-md-02 podman[2767545]: 2024-05-07 20:26:10.329094187 -0700 PDT m=+0.049135572 container remove 3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 systemd[1]: libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated successfully. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.441995116 -0700 PDT m=+0.040292335 container create c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 20:26:10 pr-md-02 systemd[1]: Started libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope. May 7 20:26:10 pr-md-02 systemd[1]: Started libcrun container. May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.475459241 -0700 PDT m=+0.073756409 container init c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482648828 -0700 PDT m=+0.080945996 container start c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.482737614 -0700 PDT m=+0.081034798 container attach c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True) May 7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07 20:26:10.426366966 -0700 PDT m=+0.024664165 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-30d82c4f42c9c8e9838529715b3cb7fd1874f71c37de5570564e1346385ea590-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231562: 1313 pgs: 5 active+clean+scrubbing+deep, 5 active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 43, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: }, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_id": 41, May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: "type": "bluestore" May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 recursing_burnell[2767581]: } May 7 20:26:11 pr-md-02 systemd[1]: libpod-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767566]: 2024-05-07 20:26:11.468853634 -0700 PDT m=+1.067150811 container died c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8b793e8aa6dce35c6bd88479a14748d9d2eeba53693b0716213fb533d5bd9a2b-merged.mount: Deactivated successfully. May 7 20:26:11 pr-md-02 podman[2767662]: 2024-05-07 20:26:11.526340084 -0700 PDT m=+0.050031409 container remove c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD) May 7 20:26:11 pr-md-02 systemd[1]: libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated successfully. May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a64027d0-50d9-4df4-84e0-2d6fc7c28d01 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9c614f97-f8ac-4106-bdc4-22a735263ae7 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 087c2c0d-6427-4c79-8604-8e8697a30ad3 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 92b105d6-1f55-4259-8118-6a5ff6cbf800 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 37ac8996-faa0-47bc-8584-d323f77d7b3f does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev f8b7279c-f34c-4451-9400-f78a5674f672 does not exist May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_03:26:11 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_metadata', '.mgr', 'cephfs_data', 'cephfs_md_and_data'] May 7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 20:26:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231563: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:13 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231564: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon) e5 handle_command mon_command({"prefix": "status"} v 0) v1 May 7 20:26:15 pr-md-02 ceph-mon[5348]: log_channel(audit) log [DBG] : from='client.? 10.50.1.74:0/2874616819' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch May 7 20:26:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:03:26:16] "GET /metrics HTTP/1.1" 200 73366 "" "Prometheus/2.43.0" May 7 20:26:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231565: 1313 pgs: 4 active+clean+scrubbing+deep, 7 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:18 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.289994 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231566: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 20:26:21 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231567: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228921. Immutable memtables: 0. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.115346) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12422] Flushing memtable with next log file: 228921 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782115376, "job": 12422, "event": "flush_started", "num_memtables": 1, "num_entries": 3185, "num_deletes": 724, "total_data_size": 2772294, "memory_usage": 2834928, "flush_reason": "Manual Compaction"} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12422] Level-0 flush table #228922: started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782128750, "cf_name": "default", "job": 12422, "event": "table_file_creation", "file_number": 228922, "file_size": 1762231, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 142008149, "largest_seqno": 142011329, "table_properties": {"data_size": 1750535, "index_size": 6357, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4293, "raw_key_size": 43713, "raw_average_key_size": 25, "raw_value_size": 1720798, "raw_average_value_size": 1017, "num_data_blocks": 243, "num_entries": 1692, "num_filter_entries": 1692, "num_deletions": 724, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715138637, "oldest_key_time": 1715138637, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228922, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12422] Flush lasted 13451 microseconds, and 5016 cpu microseconds. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128796) [db/flush_job.cc:967] [default] [JOB 12422] Level-0 flush table #228922: 1762231 bytes OK May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.128813) [db/memtable_list.cc:519] [default] Level-0 commit table #228922 started May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131803) [db/memtable_list.cc:722] [default] Level-0 commit table #228922: memtable #1 done May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131858) EVENT_LOG_v1 {"time_micros": 1715138782131834, "job": 12422, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.131894) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12422] Try to delete WAL files size 2755983, prev total WAL file size 2755983, number of live WAL files 2. May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228918.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132654) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313539323235' seq:72057594037927935, type:22 .. '7061786F73003130313539343737' seq:0, type:0; will stop at (end) May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12423] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12422 Base level 0, inputs: [228922(1720KB)], [228920(30MB)] May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782132682, "job": 12423, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228922], "files_L6": [228920], "score": -1, "input_data_size": 33301420, "oldest_snapshot_seqno": -1} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12423] Generated table #228923: 24931 keys, 32057163 bytes, temperature: kUnknown May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782250508, "cf_name": "default", "job": 12423, "event": "table_file_creation", "file_number": 228923, "file_size": 32057163, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 31912742, "index_size": 81030, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62341, "raw_key_size": 693222, "raw_average_key_size": 27, "raw_value_size": 31478263, "raw_average_value_size": 1262, "num_data_blocks": 2833, "num_entries": 24931, "num_filter_entries": 24931, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715138782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228923, "seqno_to_time_mapping": "N/A"}} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.250767) [db/compaction/compaction_job.cc:1663] [default] [JOB 12423] Compacted 1@0 + 1@6 files to L6 => 32057163 bytes May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253022) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 282.5 rd, 271.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.7, 30.1 +0.0 blob) out(30.6 +0.0 blob), read-write-amplify(37.1) write-amplify(18.2) OK, records in: 26401, records dropped: 1470 output_compression: NoCompression May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.253076) EVENT_LOG_v1 {"time_micros": 1715138782253053, "job": 12423, "event": "compaction_finished", "compaction_time_micros": 117880, "compaction_time_cpu_micros": 56129, "output_level": 6, "num_output_files": 1, "total_output_size": 32057163, "num_input_records": 26401, "num_output_records": 24931, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228922.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782253451, "job": 12423, "event": "table_file_deletion", "file_number": 228922} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228920.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715138782256177, "job": 12423, "event": "table_file_deletion", "file_number": 228920} May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.132618) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256209) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256215) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256218) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256221) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-03:26:22.256225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 20:26:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 20:26:23 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v231568: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 20:26:23 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 20:26:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata
On 5/7/24 6:05 PM, Erich Weiler wrote:
Actually looking again, I don't see core dumps (though I could have sworn I did before). But looking at my MDS servers, they appear to be in a constant cycle of trying to restart, and not making it, then trying again, etc. The logs look like what I sent in my previous email, over and over again.
I tried restarting all three of my ceph-mds daemons at the same time, and it still doesn't seem to help.
I hope I can recover...
-erich
On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,
We just noticed that our cephfs filesystem was unreachable and we saw this:
# ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w)
data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep
io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr
[root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more
And filesystem status is:
# ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there:
# tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed [Text gekürzt]
On 5/8/24 09:05, Erich Weiler wrote:
Actually looking again, I don't see core dumps (though I could have sworn I did before). But looking at my MDS servers, they appear to be in a constant cycle of trying to restart, and not making it, then trying again, etc. The logs look like what I sent in my previous email, over and over again.
I tried restarting all three of my ceph-mds daemons at the same time, and it still doesn't seem to help.
I didn't see any mds related logs. Could you catch more ? If not then could you try to enable the debug logs if you are sure the mds daemons are in a constant cycle of trying to restart ? Let's see what exactly it is trying to do and where is stuck. Thanks
I hope I can recover...
-erich
On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,
We just noticed that our cephfs filesystem was unreachable and we saw this:
# ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available
services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w)
data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep
io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr
[root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more
And filesystem status is:
# ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there:
# tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225254: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225255: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225256: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:26] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225257: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225258: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225259: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225260: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225261: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:36] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225262: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225263: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225264: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225265: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225266: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:46] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:06:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225267: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225268: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225269: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225270: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225271: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:56] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225272: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225273: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228438. Immutable memtables: 0. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.708981) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12100] Flushing memtable with next log file: 228438 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819709068, "job": 12100, "event": "flush_started", "num_memtables": 1, "num_entries": 3587, "num_deletes": 921, "total_data_size": 2910092, "memory_usage": 2992096, "flush_reason": "Manual Compaction"} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12100] Level-0 flush table #228439: started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819721834, "cf_name": "default", "job": 12100, "event": "table_file_creation", "file_number": 228439, "file_size": 1847607, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141701983, "largest_seqno": 141705565, "table_properties": {"data_size": 1834448, "index_size": 6860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5253, "raw_key_size": 55412, "raw_average_key_size": 26, "raw_value_size": 1798552, "raw_average_value_size": 857, "num_data_blocks": 265, "num_entries": 2097, "num_filter_entries": 2097, "num_deletions": 921, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126677, "oldest_key_time": 1715126677, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228439, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12100] Flush lasted 12882 microseconds, and 5541 cpu microseconds. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721870) [db/flush_job.cc:967] [default] [JOB 12100] Level-0 flush table #228439: 1847607 bytes OK May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721889) [db/memtable_list.cc:519] [default] Level-0 commit table #228439 started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724483) [db/memtable_list.cc:722] [default] Level-0 commit table #228439: memtable #1 done May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724523) EVENT_LOG_v1 {"time_micros": 1715126819724510, "job": 12100, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724561) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12100] Try to delete WAL files size 2890782, prev total WAL file size 2890782, number of live WAL files 2. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725271) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337383930' seq:72057594037927935, type:22 .. '7061786F73003130313338313432' seq:0, type:0; will stop at (end) May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12101] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12100 Base level 0, inputs: [228439(1804KB)], [228437(36MB)] May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819725328, "job": 12101, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228439], "files_L6": [228437], "score": -1, "input_data_size": 39995951, "oldest_snapshot_seqno": -1} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12101] Generated table #228440: 25379 keys, 38251292 bytes, temperature: kUnknown May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819875843, "cf_name": "default", "job": 12101, "event": "table_file_creation", "file_number": 228440, "file_size": 38251292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 38096937, "index_size": 89812, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702456, "raw_average_key_size": 27, "raw_value_size": 37648320, "raw_average_value_size": 1483, "num_data_blocks": 3163, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228440, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.876095) [db/compaction/compaction_job.cc:1663] [default] [JOB 12101] Compacted 1@0 + 1@6 files to L6 => 38251292 bytes May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878629) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 265.6 rd, 254.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.8, 36.4 +0.0 blob) out(36.5 +0.0 blob), read-write-amplify(42.4) write-amplify(20.7) OK, records in: 27250, records dropped: 1871 output_compression: NoCompression May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878669) EVENT_LOG_v1 {"time_micros": 1715126819878658, "job": 12101, "event": "compaction_finished", "compaction_time_micros": 150577, "compaction_time_cpu_micros": 64114, "output_level": 6, "num_output_files": 1, "total_output_size": 38251292, "num_input_records": 27250, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819878937, "job": 12101, "event": "table_file_deletion", "file_number": 228439} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819882652, "job": 12101, "event": "table_file_deletion", "file_number": 228437} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882737) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882741) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882743) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882744) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882746) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:07:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225274: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225275: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225276: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:06] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:07:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225277: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:07:07 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_data', 'cephfs_md_and_data', '.mgr', 'cephfs_metadata'] May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:07:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:07:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225278: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225279: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225280: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:07:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225281: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 e5614: 46 total, 46 up, 46 in May 7 17:07:15 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225283: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d807ba79-465d-4d89-ab03-9bff6275dd10 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9e26e8f6-fba9-425c-923b-ba9da32693a7 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1f435daa-e8f0-4964-b7fb-07ea14b95e9f does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.694558647 -0700 PDT m=+0.039788036 container create 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.722272344 -0700 PDT m=+0.067501679 container init 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:07:17 pr-md-02 stoic_mayer[2565752]: 167 167 May 7 17:07:17 pr-md-02 systemd[1]: libpod-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728419093 -0700 PDT m=+0.073648421 container start 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728522035 -0700 PDT m=+0.073751360 container attach 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728769341 -0700 PDT m=+0.073998662 container died 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.678321769 -0700 PDT m=+0.023551090 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:07:17 pr-md-02 podman[2565757]: 2024-05-07 17:07:17.78259101 -0700 PDT m=+0.046708660 container remove 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 systemd[1]: libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.90851263 -0700 PDT m=+0.039669496 container create 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.936966849 -0700 PDT m=+0.068123675 container init 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943702066 -0700 PDT m=+0.074858891 container start 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943794994 -0700 PDT m=+0.074951827 container attach 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.892929482 -0700 PDT m=+0.024086307 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> passed data devices: 2 physical, 0 LVM May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> relative data size: 1.0 May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> All data devices are unavailable May 7 17:07:18 pr-md-02 systemd[1]: libpod-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 podman[2565778]: 2024-05-07 17:07:18.556739188 -0700 PDT m=+0.687896012 container died 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:18 pr-md-02 podman[2565822]: 2024-05-07 17:07:18.614711286 -0700 PDT m=+0.047414226 container remove 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:18 pr-md-02 systemd[1]: libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-2f63fd9ffd2763a7ee3f75befaf99b27a75325cd025cc2e74e72d7bd150afc42-merged.mount: Deactivated successfully. May 7 17:07:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225284: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.171860525 -0700 PDT m=+0.039261474 container create eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.199519439 -0700 PDT m=+0.066920407 container init eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:19 pr-md-02 xenodochial_wu[2565974]: 167 167 May 7 17:07:19 pr-md-02 systemd[1]: libpod-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.205924433 -0700 PDT m=+0.073325392 container start eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.2060646 -0700 PDT m=+0.073465559 container attach eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.206298065 -0700 PDT m=+0.073699031 container died eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.158156321 -0700 PDT m=+0.025557273 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:07:19 pr-md-02 podman[2565979]: 2024-05-07 17:07:19.25994464 -0700 PDT m=+0.046894141 container remove eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.388083595 -0700 PDT m=+0.043236647 container create a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.417053374 -0700 PDT m=+0.072206356 container init a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423715752 -0700 PDT m=+0.078868733 container start a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423810643 -0700 PDT m=+0.078963634 container attach a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.371099332 -0700 PDT m=+0.026252317 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-91f5899d00d5794e67bbdd45fa72d22c74b0b977efec0238aa842ac272b635ce-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "41": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme0n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "41", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "43": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme1n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "43", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ] May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 systemd[1]: libpod-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.801695942 -0700 PDT m=+0.456848932 container died a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-68bc380ea9ec1cff282d08f862c7b661ba0571a20622225a5a3994e5fc88985d-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566028]: 2024-05-07 17:07:19.854983202 -0700 PDT m=+0.045986655 container remove a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.420915201 -0700 PDT m=+0.041065865 container create f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.45143613 -0700 PDT m=+0.071586769 container init f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:07:20 pr-md-02 practical_wu[2566180]: 167 167 May 7 17:07:20 pr-md-02 systemd[1]: libpod-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457513493 -0700 PDT m=+0.077664122 container start f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457638676 -0700 PDT m=+0.077789308 container attach f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457913249 -0700 PDT m=+0.078063874 container died f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.405495878 -0700 PDT m=+0.025646501 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:07:20 pr-md-02 podman[2566185]: 2024-05-07 17:07:20.515553442 -0700 PDT m=+0.045837844 container remove f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) May 7 17:07:20 pr-md-02 systemd[1]: libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.631476902 -0700 PDT m=+0.043759539 container create c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.660235834 -0700 PDT m=+0.072518401 container init c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.666915701 -0700 PDT m=+0.079198260 container start c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.667002215 -0700 PDT m=+0.079284787 container attach c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) May 7 17:07:20 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb2679257a24186947b435237af458658d281389b531640688ba39cfa7cac79-merged.mount: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.6134279 -0700 PDT m=+0.025710470 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May 7 17:07:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225285: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 43, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: }, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 41, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 systemd[1]: libpod-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566207]: 2024-05-07 17:07:21.655826716 -0700 PDT m=+1.068109291 container died c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:21 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-e573f9cf164334d9b913d82ba760fa7f00e98646ea067755c99a3842969d30d8-merged.mount: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566303]: 2024-05-07 17:07:21.711419562 -0700 PDT m=+0.047850993 container remove c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:21 pr-md-02 systemd[1]: libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 57f8febe-215e-403c-821d-95da6e3c5073 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 2637522f-3bd6-453c-976c-f7da78d12399 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b0cc7927-d150-4870-8ec7-6ef7d847ffd2 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9b053170-587d-42ee-929e-25112b3abba1 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 352cfbfb-4b0a-420c-959b-e42615f9913d does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 144dbb62-1eeb-49ba-b928-eee4a1756251 does not exist May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225286: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225287: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225288: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:29 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225289: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:31 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225290: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:33 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225291: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:35 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225292: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:37 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225293: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:39 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225294: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:41 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225295: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:43 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225296: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:45 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225297: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:46] "GET /metrics HTTP/1.1" 200 73538 "" "Prometheus/2.43.0" May 7 17:07:47 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225298: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:49 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225299: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:51 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225300: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:53 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225301: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225302: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:56] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:07:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225303: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:59 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225304: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225305: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225306: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:04 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225307: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub starts May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub ok May 7 17:08:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:06] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:08:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225308: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 e5615: 46 total, 46 up, 46 in May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:08:07 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', 'cephfs_data', '.mgr', 'cephfs_metadata'] May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:08:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:08:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225310: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:09 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:08:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225311: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225312: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated... May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:08:14 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225313: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:16] "GET /metrics HTTP/1.1" 200 73536 "" "Prometheus/2.43.0" May 7 17:08:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225314: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228441. Immutable memtables: 0. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.105603) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12102] Flushing memtable with next log file: 228441 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898105644, "job": 12102, "event": "flush_started", "num_memtables": 1, "num_entries": 2046, "num_deletes": 617, "total_data_size": 1704550, "memory_usage": 1744008, "flush_reason": "Manual Compaction"} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12102] Level-0 flush table #228442: started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898113765, "cf_name": "default", "job": 12102, "event": "table_file_creation", "file_number": 228442, "file_size": 847046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141705570, "largest_seqno": 141707611, "table_properties": {"data_size": 839933, "index_size": 2991, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 32187, "raw_average_key_size": 26, "raw_value_size": 819966, "raw_average_value_size": 670, "num_data_blocks": 115, "num_entries": 1223, "num_filter_entries": 1223, "num_deletions": 617, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126820, "oldest_key_time": 1715126820, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228442, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12102] Flush lasted 8207 microseconds, and 2745 cpu microseconds. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113810) [db/flush_job.cc:967] [default] [JOB 12102] Level-0 flush table #228442: 847046 bytes OK May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113830) [db/memtable_list.cc:519] [default] Level-0 commit table #228442 started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116583) [db/memtable_list.cc:722] [default] Level-0 commit table #228442: memtable #1 done May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116634) EVENT_LOG_v1 {"time_micros": 1715126898116612, "job": 12102, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116650) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12102] Try to delete WAL files size 1693223, prev total WAL file size 1693550, number of live WAL files 2. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228438.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.117035) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740032393030383037' seq:72057594037927935, type:22 .. '6D6772737461740032393031303630' seq:0, type:0; will stop at (end) May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12103] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12102 Base level 0, inputs: [228442(827KB)], [228440(36MB)] May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898117062, "job": 12103, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228442], "files_L6": [228440], "score": -1, "input_data_size": 39098338, "oldest_snapshot_seqno": -1} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12103] Generated table #228443: 25379 keys, 37258556 bytes, temperature: kUnknown May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898320688, "cf_name": "default", "job": 12103, "event": "table_file_creation", "file_number": 228443, "file_size": 37258556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37107602, "index_size": 86411, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702042, "raw_average_key_size": 27, "raw_value_size": 36662378, "raw_average_value_size": 1444, "num_data_blocks": 3038, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228443, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.321010) [db/compaction/compaction_job.cc:1663] [default] [JOB 12103] Compacted 1@0 + 1@6 files to L6 => 37258556 bytes May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323275) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 191.9 rd, 182.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.8, 36.5 +0.0 blob) out(35.5 +0.0 blob), read-write-amplify(90.1) write-amplify(44.0) OK, records in: 26602, records dropped: 1223 output_compression: NoCompression May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323329) EVENT_LOG_v1 {"time_micros": 1715126898323308, "job": 12103, "event": "compaction_finished", "compaction_time_micros": 203764, "compaction_time_cpu_micros": 65018, "output_level": 6, "num_output_files": 1, "total_output_size": 37258556, "num_input_records": 26602, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898323660, "job": 12103, "event": "table_file_deletion", "file_number": 228442} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898326958, "job": 12103, "event": "table_file_deletion", "file_number": 228440} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116998) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326990) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326994) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327002) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327006) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327008) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
I have not updated any packages or anything, I'm not sure how this came about. I did set the following two items earlier today, although I'm not sure if it would make any difference?
ceph config set mds mds_recall_max_caps 3000 ceph config set mds mds_cache_memory_limit 16000000000
I tried to restart the MDS daemons alike so:
ceph orch daemon restart mds.slugfs.pr-md-02.sbblqq
And they seem to start for a moment then stop again.
I then tried to restart the one remaining one that was online, then it seemed to die as well, and now I see this:
# ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
But the ceph-mds on pr-md-01 doesn't exist now.
Anyone know how to recover from this? I'm kind of panicking. :(
Thanks much, erich
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Erich Weiler
-
Eugen Block
-
Xiubo Li