ceph-users

Download

ceph-users@ceph.io

42 participants
9158 discussions

RBD Mirroring with Journaling and Snapshot mechanism
by V A Prabha 09 May '24

09 May '24

Dear Eugen, We have a scenario of DC and DR replication, and planned to explore RBD mirroring with both Journaling and Snapshot mechanism. I have a 5 TB storage at Primary DC and 5 TB storage at DR site with 2 different ceph clusters configured. Please clarify the following queries 1. With One way mirroring, the failover works fine in both journaling and snapshot mechanism and we are able to promote the workload from DR site. How does Failback work? We wanted to move the contents from DR to DC but it fails. In journaling mechanism, it deletes the entire volume and recreates it afresh which does not solve our problem. 2. How does incremental replication work from DR to DC? 3. Does Two-way mirroring help this situation. According to me, in this method, it is for 2 different clouds with 2 different storages and replicating both the clouds workloads? Does Failback work in this scenario ? Please help us / guide us to deploy this solution Regards V.A.Prabha Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

3 4

Call for Proposals: Cephalocon 2024
by Noah Lehman 09 May '24

09 May '24

Hi Ceph community, We are excited to share the CFP for Cephalocon 2024 <https://events.linuxfoundation.org/cephalocon/program/cfp/>, running December 4-5 in Geneva, Switzerland. The deadline to submit a proposal is August 11, 2024. You can submit a proposal here <https://events.linuxfoundation.org/cephalocon/program/cfp/>. Running 4-5 December 2024, Cephalocon is the premier yearly event that brings together the global community of operators, developers, and researchers to celebrate Ceph, the open source distributed storage system designed to provide excellent performance, reliability, and scalability. Join new and existing community members from around the world at CERN to learn more about Ceph and the future of the project from the developers writing the code and the operators deploying it. *DATES TO REMEMBER:* CFP Opens: 6 May, 2024 CFP Close: 11 August, 2024 Speaker Notifications: 9 September, 2024 Schedule Announced: 12 September, 2024 Slides due date: 2 December, 2024 Event Dates: 4 December, 2024 - 5 December, 2024 Best, Noah

1 0

How to handle incomplete data after rbd import-diff failure?
by Satoru Takeuchi 09 May '24

09 May '24

I confirmed that incomplete data is left on `rbd import-diff` failure. I guess that this data is the part of snapshot. Could someone answer me the following questions? Q1. Is it safe to use the RBD image (e.g. client I/O and snapshot management) even though incomplete data exists? Q2. Is there any way to clean up the incomplete data? I read the following document and understand that this problem will be resolved after running `rbd import-diff` again. https://ceph.io/en/news/blog/2013/incremental-snapshots-with-rbd/ > Since overwriting the same data is idempotent, it’s safe to have an import-diff interrupted in the middle. However, it's difficult if I can't access the exported backup data anymore. For instance, I'm afraid of the following scenario. 1. Send the backup data from one DC (DC0) to another DC (DC1) periodically. 2. The backup data is created in DC0 and is sent directly to DC1 without persist backup data as a file. 3. Major power outage happens in DC0 and it's impossible to re-generate the backup data for a long time. I simulated this problem as follows: 1. Create an RBD image. 2. Write some data to this image. 3. Create a snapshot S0. 4. Write another data to this image. 5. Create a snapshot S1. 6. Create a backup data consists of the difference between S0 and S1 by running rbd export-diff. 7. Delete the last byte of the backup data, which is 'e' and means the end of the backup data, to inject import-diff failure. 8. Delete S1. 9. Run rbd import-diff to apply the broken backup data created in the step 7. Then step9 failed and S1 was not created. However, the number of RADOS objects and the storage usage has increased. before ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 11 MiB 24 9 24 0 0 0 3609 53 MiB 279 41 MiB 0 B 0 B total_objects 24 total_used 39 MiB total_avail 32 GiB total_space 32 GiB ``` after: ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 12 MiB 25 9 25 0 0 0 3531 53 MiB 278 41 MiB 0 B 0 B total_objects 25 total_used 40 MiB total_avail 32 GiB total_space 32 GiB ``` The incomplete data seem to increase if rbd import-diff fails again and again. The following output was get after the above-mentioned step9 100 times. ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 12 MiB 25 9 25 0 0 0 7925 104 MiB 1308 164 MiB 0 B 0 B total_objects 25 total_used 58 MiB total_avail 32 GiB total_space 32 GiB ``` Thanks, Satoru

2 3

Guidance on using large RBD volumes - NTFS
by Robert W. Eckert 08 May '24

08 May '24

Hi - in my home , I have been running cephfs for a few years, and have reasonably good performance, however since exposing cephfs via SMB has been hit and miss. So I thought I could carve out space for a RBD device to share from a windows machine My set up: CEPH 18.2.2 deployed using ceph adm 4 servers running RHEL9 on AMD 5600g CPUs 64 Gb Each 10Gbe NICs 4x4Tb Hdd 1 2Tb NVME for DB/WAL rbd pool is set to auto PG - its currently at 256 I have tested the NIC connection between the servers and my PC, an each point to point works well at the 10Gbe speeds Now the problem I created a 8Tb RBD using rbd create winshare -size 8T -pool rbd rbd map winshare I prepped the drive, formatted it, and the drive appears cleanly as an 8 Tb drive. When I used fio on the drive/volume, speeds were good around 150-200 Mb/s. Then I started trying to populate the drive from a few different sources, and performance took a nose dive. - Write speeds are about 6-10 Mb/s, and windows task manager shows average response time anywhere from 500ms to 30 seconds. - mainly around 4 seconds. I don't see any obvious bottlenecks - cpu on the servers are about 5-10%, memory is good, Network is showing under 1 Gb/s on all servers. I am wondering if I needed to use different parameters for creating the volume? Or is there a practical limit to the volume size I exceeded? Thanks, Rob

1 1

cephfs filesystem offline, ceph-mds core dumped... How to recover?
by Erich Weiler 08 May '24

08 May '24

Hi All, We just noticed that our cephfs filesystem was unreachable and we saw this: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr [root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more And filesystem status is: # ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there: # tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225254: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225255: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225256: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:26] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225257: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225258: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225259: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225260: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225261: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:36] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225262: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225263: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225264: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225265: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225266: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:46] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:06:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225267: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225268: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225269: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225270: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225271: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:56] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225272: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225273: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228438. Immutable memtables: 0. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.708981) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12100] Flushing memtable with next log file: 228438 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819709068, "job": 12100, "event": "flush_started", "num_memtables": 1, "num_entries": 3587, "num_deletes": 921, "total_data_size": 2910092, "memory_usage": 2992096, "flush_reason": "Manual Compaction"} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12100] Level-0 flush table #228439: started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819721834, "cf_name": "default", "job": 12100, "event": "table_file_creation", "file_number": 228439, "file_size": 1847607, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141701983, "largest_seqno": 141705565, "table_properties": {"data_size": 1834448, "index_size": 6860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5253, "raw_key_size": 55412, "raw_average_key_size": 26, "raw_value_size": 1798552, "raw_average_value_size": 857, "num_data_blocks": 265, "num_entries": 2097, "num_filter_entries": 2097, "num_deletions": 921, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126677, "oldest_key_time": 1715126677, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228439, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12100] Flush lasted 12882 microseconds, and 5541 cpu microseconds. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721870) [db/flush_job.cc:967] [default] [JOB 12100] Level-0 flush table #228439: 1847607 bytes OK May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721889) [db/memtable_list.cc:519] [default] Level-0 commit table #228439 started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724483) [db/memtable_list.cc:722] [default] Level-0 commit table #228439: memtable #1 done May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724523) EVENT_LOG_v1 {"time_micros": 1715126819724510, "job": 12100, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724561) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12100] Try to delete WAL files size 2890782, prev total WAL file size 2890782, number of live WAL files 2. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725271) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337383930' seq:72057594037927935, type:22 .. '7061786F73003130313338313432' seq:0, type:0; will stop at (end) May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12101] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12100 Base level 0, inputs: [228439(1804KB)], [228437(36MB)] May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819725328, "job": 12101, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228439], "files_L6": [228437], "score": -1, "input_data_size": 39995951, "oldest_snapshot_seqno": -1} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12101] Generated table #228440: 25379 keys, 38251292 bytes, temperature: kUnknown May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819875843, "cf_name": "default", "job": 12101, "event": "table_file_creation", "file_number": 228440, "file_size": 38251292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 38096937, "index_size": 89812, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702456, "raw_average_key_size": 27, "raw_value_size": 37648320, "raw_average_value_size": 1483, "num_data_blocks": 3163, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228440, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.876095) [db/compaction/compaction_job.cc:1663] [default] [JOB 12101] Compacted 1@0 + 1@6 files to L6 => 38251292 bytes May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878629) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 265.6 rd, 254.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.8, 36.4 +0.0 blob) out(36.5 +0.0 blob), read-write-amplify(42.4) write-amplify(20.7) OK, records in: 27250, records dropped: 1871 output_compression: NoCompression May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878669) EVENT_LOG_v1 {"time_micros": 1715126819878658, "job": 12101, "event": "compaction_finished", "compaction_time_micros": 150577, "compaction_time_cpu_micros": 64114, "output_level": 6, "num_output_files": 1, "total_output_size": 38251292, "num_input_records": 27250, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819878937, "job": 12101, "event": "table_file_deletion", "file_number": 228439} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819882652, "job": 12101, "event": "table_file_deletion", "file_number": 228437} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882737) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882741) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882743) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882744) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882746) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:07:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225274: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225275: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225276: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:06] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:07:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225277: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:07:07 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_data', 'cephfs_md_and_data', '.mgr', 'cephfs_metadata'] May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:07:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:07:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225278: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225279: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225280: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:07:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225281: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 e5614: 46 total, 46 up, 46 in May 7 17:07:15 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225283: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d807ba79-465d-4d89-ab03-9bff6275dd10 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9e26e8f6-fba9-425c-923b-ba9da32693a7 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1f435daa-e8f0-4964-b7fb-07ea14b95e9f does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.694558647 -0700 PDT m=+0.039788036 container create 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.722272344 -0700 PDT m=+0.067501679 container init 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:07:17 pr-md-02 stoic_mayer[2565752]: 167 167 May 7 17:07:17 pr-md-02 systemd[1]: libpod-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728419093 -0700 PDT m=+0.073648421 container start 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728522035 -0700 PDT m=+0.073751360 container attach 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728769341 -0700 PDT m=+0.073998662 container died 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.678321769 -0700 PDT m=+0.023551090 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:17 pr-md-02 podman[2565757]: 2024-05-07 17:07:17.78259101 -0700 PDT m=+0.046708660 container remove 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 systemd[1]: libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.90851263 -0700 PDT m=+0.039669496 container create 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.936966849 -0700 PDT m=+0.068123675 container init 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943702066 -0700 PDT m=+0.074858891 container start 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943794994 -0700 PDT m=+0.074951827 container attach 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.892929482 -0700 PDT m=+0.024086307 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> passed data devices: 2 physical, 0 LVM May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> relative data size: 1.0 May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> All data devices are unavailable May 7 17:07:18 pr-md-02 systemd[1]: libpod-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 podman[2565778]: 2024-05-07 17:07:18.556739188 -0700 PDT m=+0.687896012 container died 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:18 pr-md-02 podman[2565822]: 2024-05-07 17:07:18.614711286 -0700 PDT m=+0.047414226 container remove 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:18 pr-md-02 systemd[1]: libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-2f63fd9ffd2763a7ee3f75befaf99b27a75325cd025cc2e74e72d7bd150afc42-merged.mount: Deactivated successfully. May 7 17:07:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225284: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.171860525 -0700 PDT m=+0.039261474 container create eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.199519439 -0700 PDT m=+0.066920407 container init eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:19 pr-md-02 xenodochial_wu[2565974]: 167 167 May 7 17:07:19 pr-md-02 systemd[1]: libpod-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.205924433 -0700 PDT m=+0.073325392 container start eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.2060646 -0700 PDT m=+0.073465559 container attach eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.206298065 -0700 PDT m=+0.073699031 container died eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.158156321 -0700 PDT m=+0.025557273 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 podman[2565979]: 2024-05-07 17:07:19.25994464 -0700 PDT m=+0.046894141 container remove eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.388083595 -0700 PDT m=+0.043236647 container create a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.417053374 -0700 PDT m=+0.072206356 container init a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423715752 -0700 PDT m=+0.078868733 container start a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423810643 -0700 PDT m=+0.078963634 container attach a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.371099332 -0700 PDT m=+0.026252317 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-91f5899d00d5794e67bbdd45fa72d22c74b0b977efec0238aa842ac272b635ce-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "41": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme0n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "41", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "43": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme1n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "43", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ] May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 systemd[1]: libpod-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.801695942 -0700 PDT m=+0.456848932 container died a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-68bc380ea9ec1cff282d08f862c7b661ba0571a20622225a5a3994e5fc88985d-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566028]: 2024-05-07 17:07:19.854983202 -0700 PDT m=+0.045986655 container remove a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.420915201 -0700 PDT m=+0.041065865 container create f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.45143613 -0700 PDT m=+0.071586769 container init f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:07:20 pr-md-02 practical_wu[2566180]: 167 167 May 7 17:07:20 pr-md-02 systemd[1]: libpod-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457513493 -0700 PDT m=+0.077664122 container start f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457638676 -0700 PDT m=+0.077789308 container attach f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457913249 -0700 PDT m=+0.078063874 container died f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.405495878 -0700 PDT m=+0.025646501 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 podman[2566185]: 2024-05-07 17:07:20.515553442 -0700 PDT m=+0.045837844 container remove f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) May 7 17:07:20 pr-md-02 systemd[1]: libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.631476902 -0700 PDT m=+0.043759539 container create c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.660235834 -0700 PDT m=+0.072518401 container init c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.666915701 -0700 PDT m=+0.079198260 container start c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.667002215 -0700 PDT m=+0.079284787 container attach c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) May 7 17:07:20 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb2679257a24186947b435237af458658d281389b531640688ba39cfa7cac79-merged.mount: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.6134279 -0700 PDT m=+0.025710470 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225285: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 43, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: }, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 41, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 systemd[1]: libpod-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566207]: 2024-05-07 17:07:21.655826716 -0700 PDT m=+1.068109291 container died c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:21 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-e573f9cf164334d9b913d82ba760fa7f00e98646ea067755c99a3842969d30d8-merged.mount: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566303]: 2024-05-07 17:07:21.711419562 -0700 PDT m=+0.047850993 container remove c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:21 pr-md-02 systemd[1]: libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 57f8febe-215e-403c-821d-95da6e3c5073 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 2637522f-3bd6-453c-976c-f7da78d12399 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b0cc7927-d150-4870-8ec7-6ef7d847ffd2 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9b053170-587d-42ee-929e-25112b3abba1 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 352cfbfb-4b0a-420c-959b-e42615f9913d does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 144dbb62-1eeb-49ba-b928-eee4a1756251 does not exist May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225286: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225287: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225288: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:29 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225289: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:31 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225290: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:33 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225291: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:35 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225292: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:37 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225293: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:39 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225294: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:41 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225295: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:43 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225296: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:45 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225297: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:46] "GET /metrics HTTP/1.1" 200 73538 "" "Prometheus/2.43.0" May 7 17:07:47 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225298: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:49 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225299: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:51 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225300: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:53 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225301: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225302: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:56] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:07:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225303: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:59 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225304: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225305: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225306: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:04 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225307: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub starts May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub ok May 7 17:08:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:06] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:08:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225308: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 e5615: 46 total, 46 up, 46 in May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:08:07 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', 'cephfs_data', '.mgr', 'cephfs_metadata'] May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:08:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:08:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225310: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:09 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:08:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225311: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225312: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:08:14 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225313: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:16] "GET /metrics HTTP/1.1" 200 73536 "" "Prometheus/2.43.0" May 7 17:08:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225314: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228441. Immutable memtables: 0. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.105603) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12102] Flushing memtable with next log file: 228441 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898105644, "job": 12102, "event": "flush_started", "num_memtables": 1, "num_entries": 2046, "num_deletes": 617, "total_data_size": 1704550, "memory_usage": 1744008, "flush_reason": "Manual Compaction"} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12102] Level-0 flush table #228442: started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898113765, "cf_name": "default", "job": 12102, "event": "table_file_creation", "file_number": 228442, "file_size": 847046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141705570, "largest_seqno": 141707611, "table_properties": {"data_size": 839933, "index_size": 2991, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 32187, "raw_average_key_size": 26, "raw_value_size": 819966, "raw_average_value_size": 670, "num_data_blocks": 115, "num_entries": 1223, "num_filter_entries": 1223, "num_deletions": 617, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126820, "oldest_key_time": 1715126820, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228442, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12102] Flush lasted 8207 microseconds, and 2745 cpu microseconds. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113810) [db/flush_job.cc:967] [default] [JOB 12102] Level-0 flush table #228442: 847046 bytes OK May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113830) [db/memtable_list.cc:519] [default] Level-0 commit table #228442 started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116583) [db/memtable_list.cc:722] [default] Level-0 commit table #228442: memtable #1 done May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116634) EVENT_LOG_v1 {"time_micros": 1715126898116612, "job": 12102, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116650) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12102] Try to delete WAL files size 1693223, prev total WAL file size 1693550, number of live WAL files 2. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228438.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.117035) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740032393030383037' seq:72057594037927935, type:22 .. '6D6772737461740032393031303630' seq:0, type:0; will stop at (end) May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12103] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12102 Base level 0, inputs: [228442(827KB)], [228440(36MB)] May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898117062, "job": 12103, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228442], "files_L6": [228440], "score": -1, "input_data_size": 39098338, "oldest_snapshot_seqno": -1} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12103] Generated table #228443: 25379 keys, 37258556 bytes, temperature: kUnknown May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898320688, "cf_name": "default", "job": 12103, "event": "table_file_creation", "file_number": 228443, "file_size": 37258556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37107602, "index_size": 86411, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702042, "raw_average_key_size": 27, "raw_value_size": 36662378, "raw_average_value_size": 1444, "num_data_blocks": 3038, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228443, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.321010) [db/compaction/compaction_job.cc:1663] [default] [JOB 12103] Compacted 1@0 + 1@6 files to L6 => 37258556 bytes May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323275) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 191.9 rd, 182.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.8, 36.5 +0.0 blob) out(35.5 +0.0 blob), read-write-amplify(90.1) write-amplify(44.0) OK, records in: 26602, records dropped: 1223 output_compression: NoCompression May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323329) EVENT_LOG_v1 {"time_micros": 1715126898323308, "job": 12103, "event": "compaction_finished", "compaction_time_micros": 203764, "compaction_time_cpu_micros": 65018, "output_level": 6, "num_output_files": 1, "total_output_size": 37258556, "num_input_records": 26602, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898323660, "job": 12103, "event": "table_file_deletion", "file_number": 228442} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898326958, "job": 12103, "event": "table_file_deletion", "file_number": 228440} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116998) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326990) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326994) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327002) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327006) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327008) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting I have not updated any packages or anything, I'm not sure how this came about. I did set the following two items earlier today, although I'm not sure if it would make any difference? ceph config set mds mds_recall_max_caps 3000 ceph config set mds mds_cache_memory_limit 16000000000 I tried to restart the MDS daemons alike so: ceph orch daemon restart mds.slugfs.pr-md-02.sbblqq And they seem to start for a moment then stop again. I then tried to restart the one remaining one that was online, then it seemed to die as well, and now I see this: # ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) But the ceph-mds on pr-md-01 doesn't exist now. Anyone know how to recover from this? I'm kind of panicking. :( Thanks much, erich

3 6

How to define a read-only sub-user?
by Matthew Darwin 07 May '24

07 May '24

Hi, I'm new to bucket policies. I'm trying to create a sub-user that has only read-only access to all the buckets of the main user. I created the below policy, I can't create or delete files, but I can still create buckets using "rclone mkdir". Any idea what I'm doing wrong? I'm using ceph quincy. radosgw-admin subuser create --uid=main_user --subuser=eosn_read --access=read radosgw-admin key create --subuser=main_user:sub_user --key-type=s3 --access-key xxxx --secret-key xxxx s3cmd setpolicy policy.txt s3://somebucket { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/main_user:sub_user" ] }, "Action": [ "s3:ListBucket", "s3:ListAllMyBuckets", "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectTagging", "s3:GetObjectRetention", "s3:GetObjectLegalHold" ], "Resource": "arn:aws:s3:::*" }

1 0

Removed host in maintenance mode
by Johan 07 May '24

07 May '24

Hi all, In my small cluster of 6 hosts I had troubles with a host (osd:s) and was planning to remove it from the cluster. Before I got to do that I needed to power down this host and therefore put it in maintenance mode. Due to some mistakes on my part I couldn't boot the host again and simply decided to force the removal from the cluster. The host is now removed but ceph (17.2.7) keep on complaining on it being in maintenance mode. How can I remove the last remnants of this host from the cluster? /Johan

2 3

cephadm upgrade: heartbeat failures not considered
by Eugen Block 07 May '24

07 May '24

Hi, we're facing an issue during upgrades (and sometimes server reboots), it appears to occur when (at leat) one of the MONs has to do a full sync. And I'm wondering if the upgrade procedure could be improved in that regard, I'll come back to that later. First, I'll try to summarize the events. We upgraded to latest Pacific (16.2.15) last week. Note that the filesystems of the MONs are on HDDs (we're planning to move to flash, we already noticed issues during disaster recovery on HDDs last year). According to the logs the MGRs and MONs were upgraded successfully and a quorum was formed successfully at: 2024-04-30T12:57:22.347182+0000 mon.ndeceph03 (mon.0) 1208046 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum ndeceph03,ndeceph01) Then ceph started the first OSD upgrades a few minutes later, the first one at: 2024-04-30T13:00:42.733528+0000 mon.ndeceph03 (mon.0) 101 : cluster [INF] osd.25 marked itself down and dead One OSD by one on ndeceph01 was upgraded: 2024-04-30T13:01:46.517+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.19 (5/7) 2024-04-30T13:01:46.517+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.19 (5/7) 2024-04-30T13:02:24.800+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.22 (6/7) 2024-04-30T13:02:24.800+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.22 (6/7) 2024-04-30T13:02:48.220+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.29 (7/7) 2024-04-30T13:02:48.220+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.29 (7/7) But the mon service on ndeceph02 (also OSD server) was still syncing (for around 6 minutes): 2024-04-30T13:02:33.124+0000 7f1c24444700 1 mon.ndeceph02@2(synchronizing) e58 handle_auth_request failed to assign global_id 2024-04-30T13:08:30.123+0000 7f1c24444700 1 mon.ndeceph02@2(synchronizing) e58 handle_auth_request failed to assign global_id All HDD OSDs from ndeceph02 (not upgraded yet) were complaining about timeouts, and apparently tried to reboot multiple times (wondering why the SSDs didn't complain though): 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f2e1399b700' had timed out after 15.000000954s 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 osd.0 367411 is_healthy false -- internal heartbeat failed 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 osd.0 367411 not healthy; waiting to boot 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f2e1399b700' had timed out after 15.000000954s 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 osd.0 367411 is_healthy false -- internal heartbeat failed 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 osd.0 367411 not healthy; waiting to boot ... 2024-04-30T13:03:59.322+0000 7f2e201b4700 1 osd.0 367414 state: booting -> active 2024-04-30T13:05:22.184+0000 7f2e201b4700 1 osd.0 367433 state: booting -> active 2024-04-30T13:06:39.602+0000 7f2e201b4700 1 osd.0 367447 state: booting -> active 2024-04-30T13:08:02.612+0000 7f2e201b4700 1 osd.0 367454 state: booting -> active During this syncing period the disk utilization of the OS filesystem was at 100%, but ceph kept upgrading other OSD daemons. There are 3 main hosts, failure domain is host, replicated pools with min_size 2, size 3. If the OSDs from one host are struggling (because of disk IO on the filesystem, apparently) and ceph keeps upgrading others, we get inactive PGs. So what I'm wondering about is, how does the orchestrator decide if it's ok to stop one OSD ('ceph osd ok-to-stop <ID>' is the manual command) while others obviously are not healthy and would cause inactive PGs? The cluster did notice slow requests and reported messages like these: 2024-04-30T13:02:58.867499+0000 osd.28 (osd.28) 10529 : cluster [WRN] Monitor daemon marked osd.28 down, but it is still running If some OSDs clearly aren't healthy, I would expect the orchestrator to pause the upgrade. After the mon full sync completed, it started upgrading the OSDs on ndeceph02 as well, but the inactive PGs were only resolved after several more minutes when almost all OSDs from that tree branch had been upgraded: 2024-04-30T13:16:17.536359+0000 mon.ndeceph03 (mon.0) 1461 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 11 pgs peering) During a planned upgrade this issue can be mitigated via staggered upgrades (now that I know what the cause is), upgrading MGRs and MONs first and wait until everything has settled. Then continue with OSDs. But after a reboot there's no way to control that, of course. I helped a customer last year with a mon sync issue so we might be able to improve things a bit until we have flash disks. I don't know how it all behaves in newer ceph versions regarding the upgrade, but I feel like it wouldn't make a difference right now. Is there any way to improve the orchestrator to consider unhealthy OSDs before stopping healthy ones? Thanks! Eugen

1 0

cache pressure?
by Erich Weiler 07 May '24

07 May '24

So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure". Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time? I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...? We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping... -erich

3 9

Dashboard issue slowing to a crawl - active ceph mgr process spiking to 600%+
by Zachary Perry 07 May '24

07 May '24

Hello All, I'm hoping I can get some help with an issue in the dashboard after doing a recent bare metal ceph upgrade from Octopus to Quincy. ** Please note, this document references it only being an issue with the images tab shortly after this I found the same issue on another cluster that was recently upgraded from Octopus to quincy 17.2.7 within the last few months and it's affecting all tabs in the ceph dashboard it slows to a crawl until I restart or fail over the mgr both running on top of ubuntu 20.04 Everything appears to be working fine besides the Block --> images tab. It doesn't matter what node I fail over to, reboots, reinstalling ceph-mgr-dashboard, different broswers, clients etc It will not load the 4 RBDs I have, they appear in rbd ls, I can query them and the connection on the end appliance is fine. The loading icons spin infinitely without any failure message. If I access the images tab and then move to any other tab is the dashboard it will allow me to navigate but not display anything until I either restart the service on the active mgr or fail over to another, so it works as expected until I access this one tab. when I use any other section in the in the dashboard cpu utilization for the ceph-mgr is normal but when I access the images tab it's spiked to as high as 600% and will stay like that until I restart the service or fail over the active mgr -- Active MGR before clicking Block, the OSDs spike for a second but revert to around 5% top - 13:43:37 up 8 days, 23:09, 1 user, load average: 8.08, 5.02, 4.37 Tasks: 695 total, 1 running, 694 sleeping, 0 stopped, 0 zombie %Cpu(s): 7.0 us, 1.6 sy, 0.0 ni, 89.7 id, 1.2 wa, 0.0 hi, 0.5 si, 0.0 st ----- MiB Mem : 128474.1 total, 6705.6 free, 65684.0 used, 56084.5 buff/cache MiB Swap: 40927.0 total, 35839.3 free, 5087.7 used. 49253.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14156 ceph 20 0 3420632 1.9g 13668 S 55.3 1.5 864:49.51 ceph-osd 13762 ceph 20 0 3421384 1.8g 13432 S 51.3 1.4 960:22.12 ceph-osd 14163 ceph 20 0 3422352 1.7g 13016 S 50.0 1.3 902:41.19 ceph-osd 13803 ceph 20 0 3469596 1.8g 13532 S 44.7 1.4 941:55.10 ceph-osd 13774 ceph 20 0 3427560 1.7g 13656 S 38.7 1.4 932:02.51 ceph-osd 13801 ceph 20 0 3439796 1.7g 13448 S 37.7 1.3 981:25.55 ceph-osd 14025 ceph 20 0 3426360 1.8g 13780 S 36.4 1.4 994:00.75 ceph-osd 9888 nobody 20 0 126100 8696 0 S 21.2 0.0 1106:19 node_exporter 126798 ceph 20 0 1787824 528000 39464 S 7.9 0.4 0:14.84 ceph-mgr 13795 ceph 20 0 3420252 1.7g 13264 S 7.6 1.4 990:00.61 ceph-osd 13781 ceph 20 0 3484476 1.9g 13248 S 6.3 1.5 1040:10 ceph-osd 13777 ceph 20 0 3408972 1.8g 13464 S 6.0 1.5 1026:21 ceph-osd 13797 ceph 20 0 3432068 1.6g 13932 S 6.0 1.3 950:39.35 ceph-osd 13779 ceph 20 0 3471668 1.7g 12728 S 5.6 1.3 984:53.80 ceph-osd 13768 ceph 20 0 3496064 1.9g 13504 S 5.3 1.5 918:37.48 ceph-osd 13786 ceph 20 0 3422044 1.6g 13456 S 5.3 1.3 974:29.08 ceph-osd 13788 ceph 20 0 3454184 1.9g 13048 S 5.3 1.5 980:35.78 ceph-osd 13776 ceph 20 0 3445680 1.7g 12880 S 5.0 1.3 998:30.58 ceph-osd 13785 ceph 20 0 3409548 1.7g 13704 S 5.0 1.3 939:37.08 ceph-osd 14152 ceph 20 0 3465284 1.7g 13840 S 5.0 1.4 959:39.42 ceph-osd 10339 nobody 20 0 6256048 531428 60188 S 4.6 0.4 239:37.56 prometheus 13802 ceph 20 0 3430696 1.8g 13872 S 4.6 1.4 924:15.74 ceph-osd 13791 ceph 20 0 3498876 1.5g 12648 S 4.3 1.2 962:58.37 ceph-osd 13800 ceph 20 0 3455268 1.7g 12404 S 4.3 1.3 1000:41 ceph-osd 13790 ceph 20 0 3434364 1.6g 13516 S 3.3 1.3 974:16.46 ceph-osd 14217 ceph 20 0 3443436 1.8g 13560 S 3.3 1.4 902:54.22 ceph-osd 13526 ceph 20 0 1012048 499628 11244 S 3.0 0.4 349:35.28 ceph-mon 13775 ceph 20 0 3367284 1.6g 13940 S 3.0 1.3 878:38.27 ceph-osd 13784 ceph 20 0 3380960 1.8g 12892 S 3.0 1.4 910:50.47 ceph-osd 13789 ceph 20 0 3432876 1.6g 12464 S 2.6 1.2 922:45.15 ceph-osd 13804 ceph 20 0 3428120 1.9g 13192 S 2.6 1.5 865:31.30 ceph-osd 14153 ceph 20 0 3432752 1.8g 12576 S 2.3 1.4 874:27.92 ceph-osd 14192 ceph 20 0 3412640 1.9g 13512 S 2.3 1.5 923:01.97 ceph-osd 13796 ceph 20 0 3433016 1.8g 13164 S 2.0 1.4 982:08.21 ceph-osd 13798 ceph 20 0 3405708 1.6g 13508 S 2.0 1.3 873:50.34 ceph-osd 13814 ceph 20 0 4243252 1.5g 13500 S 2.0 1.2 2020:41 ceph-osd 13985 ceph 20 0 3487848 1.6g 13100 S 2.0 1.3 942:21.96 ceph-osd 14001 ceph 20 0 4194336 1.9g 13460 S 2.0 1.5 2143:46 ceph-osd 14186 ceph 20 0 3441852 1.5g 13360 S 2.0 1.2 956:30.81 ceph-osd 7257 root 20 0 82332 3480 2984 S 0.3 0.0 9:22.50 irqbalance 7269 syslog 20 0 224344 3648 2392 S 0.3 0.0 0:11.79 rsyslogd 16621 472 20 0 898376 79800 11080 S 0.3 0.1 146:36.08 grafana 104366 root 20 0 0 0 0 I 0.3 0.0 1:11.30 kworker/0:2- events 125676 root 20 0 0 0 0 I 0.3 0.0 0:00.48 kworker/u104:7- public-bond 127115 root 20 0 10172 4636 3392 R 0.3 0.0 0:00.10 top 1 root 20 0 180712 16312 5652 S 0.0 0.0 1:31.85 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.15 kthreadd --- top - 13:44:28 up 8 days, 23:09, 1 user, load average: 6.79, 5.11, 4.43 Tasks: 695 total, 1 running, 694 sleeping, 0 stopped, 0 zombie %Cpu(s): 12.9 us, 2.1 sy, 0.0 ni, 83.5 id, 1.0 wa, 0.0 hi, 0.5 si, 0.0 st MiB Mem : 128474.1 total, 6219.8 free, 66504.7 used, 55749.6 buff/cache MiB Swap: 40927.0 total, 35837.0 free, 5090.0 used. 48435.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 126798 ceph 20 0 1854596 573432 41484 S 482.5 0.4 1:45.70 ceph-mgr 14156 ceph 20 0 3420632 1.9g 13668 S 54.6 1.5 865:16.58 ceph-osd 13762 ceph 20 0 3421384 1.9g 13432 S 51.7 1.5 960:47.47 ceph-osd 13803 ceph 20 0 3469596 1.9g 13532 S 49.3 1.5 942:18.23 ceph-osd 14163 ceph 20 0 3422352 1.9g 13016 S 49.3 1.5 903:05.84 ceph-osd 13795 ceph 20 0 3420252 1.8g 13264 S 7.0 1.4 990:04.41 ceph-osd 13777 ceph 20 0 3408972 1.9g 13464 S 5.6 1.5 1026:24 ceph-osd 13786 ceph 20 0 3422044 1.6g 13456 S 5.3 1.3 974:32.09 ceph-osd 13797 ceph 20 0 3432068 1.6g 13932 S 5.3 1.3 950:42.02 ceph-osd 16621 472 20 0 898376 78776 11044 S 5.3 0.1 146:36.76 grafana 13791 ceph 20 0 3498876 1.5g 12648 S 5.0 1.2 963:00.94 ceph-osd 14001 ceph 20 0 4194336 1.9g 13460 S 5.0 1.5 2143:47 ceph-osd 9888 nobody 20 0 126100 8696 0 S 4.6 0.0 1106:23 node_exporter 13768 ceph 20 0 3496064 1.9g 13504 S 4.6 1.5 918:39.85 ceph-osd 13776 ceph 20 0 3445680 1.7g 12880 S 4.6 1.3 998:32.98 ceph-osd 13781 ceph 20 0 3484476 1.6g 13248 S 4.6 1.3 1040:13 ceph-osd 13785 ceph 20 0 3409548 1.7g 13704 S 4.6 1.3 939:40.08 ceph-osd 13788 ceph 20 0 3454184 1.9g 13048 S 4.6 1.5 980:38.24 ceph-osd 13779 ceph 20 0 3471668 1.7g 12728 S 4.3 1.3 984:56.39 ceph-osd 13800 ceph 20 0 3455268 1.7g 12404 S 4.3 1.3 1000:44 ceph-osd 13802 ceph 20 0 3430696 1.8g 13872 S 4.0 1.4 924:18.14 ceph-osd 14152 ceph 20 0 3465284 1.7g 13840 S 4.0 1.4 959:41.83 ceph-osd 13796 ceph 20 0 3433016 1.8g 13164 S 3.0 1.4 982:09.66 ceph-osd 13784 ceph 20 0 3380960 1.8g 12892 S 2.6 1.4 910:52.06 ceph-osd 13790 ceph 20 0 3434364 1.6g 13516 S 2.6 1.3 974:17.99 ceph-osd 13801 ceph 20 0 3439796 1.9g 13448 S 2.6 1.5 981:42.61 ceph-osd 14153 ceph 20 0 3432752 1.8g 12576 S 2.6 1.4 874:29.30 ceph-osd 14186 ceph 20 0 3441852 1.6g 13360 S 2.6 1.2 956:32.32 ceph-osd 14192 ceph 20 0 3412640 1.9g 13512 S 2.6 1.5 923:03.59 ceph-osd 13526 ceph 20 0 1012048 496332 11208 S 2.3 0.4 349:36.89 ceph-mon 13789 ceph 20 0 3432876 1.6g 12464 S 2.3 1.2 922:46.59 ceph-osd 13798 ceph 20 0 3405708 1.6g 13508 S 2.3 1.3 873:51.73 ceph-osd 14217 ceph 20 0 3443436 1.8g 13560 S 2.3 1.4 902:55.74 ceph-osd 13774 ceph 20 0 3427560 1.9g 13656 S 2.0 1.5 932:19.26 ceph-osd 13775 ceph 20 0 3367284 1.6g 13940 S 2.0 1.3 878:39.76 ceph-osd 13814 ceph 20 0 4243252 1.5g 13500 S 2.0 1.2 2020:42 ceph-osd 13985 ceph 20 0 3487848 1.6g 13100 S 2.0 1.3 942:23.39 ceph-osd 13804 ceph 20 0 3428120 1.9g 13192 S 1.7 1.5 865:32.71 ceph-osd 14025 ceph 20 0 3426360 1.8g 13780 S 1.7 1.5 994:17.32 ceph-osd 10339 nobody 20 0 6256048 537184 60136 S 1.0 0.4 239:38.44 prometheus 17547 nobody 20 0 128448 8572 0 S 1.0 0.0 31:54.99 alertmanager 127115 root 20 0 10172 4636 3392 R 0.7 0.0 0:00.43 top --- OS: Ubuntu 20.04 128GB of memory Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz X11SPL-F --- Ceph Versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 140 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "ctdb": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 1 }, "rgw": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 154 } } --- cluster: id: 388dda42-9dd0-4858-a978-b3dc4c3b9152 health: HEALTH_OK services: mon: 3 daemons, quorum jarn29,jarn30,jarn31 (age 8d) mgr: osd31(active, since 10m), standbys: osd30, osd29, osd32 mds: 2/2 daemons up, 2 standby osd: 140 osds: 140 up (since 3d), 140 in (since 3d) flags noautoscale ctdb: 1 daemon active (1 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 19 pools, 5024 pgs objects: 792.78M objects, 865 TiB usage: 1.4 PiB used, 586 TiB / 2.0 PiB avail pgs: 5013 active+clean 11 active+clean+scrubbing+deep io: client: 19 MiB/s rd, 102 MiB/s wr, 672 op/s rd, 351 op/s wr --- I've ran perf and included the ceph-mgr.log for the first system that is displaying the issue Perf - https://imgur.com/a/VMh4tDf mgr log while accessing RBD tab - https://pastebin.com/t96WCWfc mgr logs prior to clicking RBD -https://pastebin.com/e4dtuD3i --- Apologies for the formatting first time posting here If anything else is needed please let me know!

2 1

Jump to page: