May 2024 - ceph-users - lists.ceph.io

Reef: Dashboard: Object Gateway Graphs have no Data
by Dave Hall 10 May '24

10 May '24

Hello. We're running a containerized deployment of Reef with a focus on RGW. We noticed that while the Grafana graphs for other categories - OSDs, Pools, etc - have data, the graphs for the Object Gateway category are empty. I did some looking last week and found reference to something about an unintended limit on the number of graphs that could be rendered. However, I did not find reference to a fix. Is it a setting that needs to be changed, or is it a piece of code that is pending for the next fix release? A second - possibly related question: There is an informational message on the Object Gateway panel about how the default realm is not set. We're not using multi-site sync, so we haven't set the realm, but we're wondering if we should set a default realm just to keep the code happy. We're also wondering if this is related in any way to the empty graphs described above. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

2 1

MDS crash in interval_set: FAILED ceph_assert(p->first <= start)
by Dejan Lesjak 10 May '24

10 May '24

Hello, We have cephfs with two active MDS. Currently rank 1 is repeatedly crashing with FAILED ceph_assert(p->first <= start) in md_log_replay thread. Is there any way to work around this and get to accesible file system or should we start with disaster recovery? It seems similar to https://tracker.ceph.com/issues/61009 Crash info: { "assert_condition": "p->first <= start", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h", "assert_func": "void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]", "assert_line": 568, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7fcdaaf8a640 time 2024-05-08T00:26:22.049974+0200\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el9/BUILD/ceph-18.2.2/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start)\n", "assert_thread_name": "md_log_replay", "backtrace": [ "/lib64/libc.so.6(+0x54db0) [0x7fcdb7a54db0]", "/lib64/libc.so.6(+0xa154c) [0x7fcdb7aa154c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcdb83610ff]", "/usr/lib64/ceph/libceph-common.so.2(+0x161263) [0x7fcdb8361263]", "/usr/bin/ceph-mds(+0x1f3b0e) [0x55a5904a9b0e]", "/usr/bin/ceph-mds(+0x1f3b55) [0x55a5904a9b55]", "(EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4b9d) [0x55a5906e1c8d]", "(EUpdate::replay(MDSRank*)+0x5d) [0x55a5906eacbd]", "(MDLog::_replay_thread()+0x7a1) [0x55a590694af1]", "/usr/bin/ceph-mds(+0x1460f1) [0x55a5903fc0f1]", "/lib64/libc.so.6(+0x9f802) [0x7fcdb7a9f802]", "/lib64/libc.so.6(+0x3f450) [0x7fcdb7a3f450]" ], "ceph_version": "18.2.2", "crash_id": "2024-05-07T22:26:22.050652Z_8be89ffb-bb87-4832-9339-57f8bd29f766", "entity_name": "mds.spod19", "os_id": "almalinux", "os_name": "AlmaLinux", "os_version": "9.3 (Shamrock Pampas Cat)", "os_version_id": "9.3", "process_name": "ceph-mds", "stack_sig": "3d0a2ca9b3c7678bf69efc20fff42b588c63f8be1832e1e0c28c99bafc082c15", "timestamp": "2024-05-07T22:26:22.050652Z", "utsname_hostname": "spod19.ijs.si", "utsname_machine": "x86_64", "utsname_release": "5.14.0-362.8.1.el9_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Tue Nov 7 14:54:22 EST 2023" } Cheers, Dejan

3 5

Forcing Posix Permissions On New CephFS Files
by duluxoz 10 May '24

10 May '24

Hi All, I've gone and gotten myself into a "can't see the forest for the trees" state, so I'm hoping someone can take pity on me and answer a really dumb Q. So I've got a CephFS system happily bubbling along and a bunch of (linux) workstations connected to a number of common shares/folders. To take a single one of these folders as an example ("music") the sub-folders and files of that share all belong to root:music with permissions of 2770 (folders) and 0660 (files). The "music" folder is then connected to (as per the Ceph Doco: mount.ceph) via each workstation's fstab file - all good, all working, everyone's happy. What I'm trying to achieve is that when a new piece of music (a file) is uploaded to the Ceph Cluster the file inherits the music share's default ownership (root:music) and permissions (0660). What is happening at the moment is I'm getting permissions of 644 (and 755 for new folders). I've been looking for a way to do what I want but, as I said, I've gone and gotten myself thoroughly mixed-up. Could someone please point me in the right direction on how to achieve what I'm after - thanks Cheers Dulux-Oz

3 2

RBD Mirroring with Journaling and Snapshot mechanism
by V A Prabha 09 May '24

09 May '24

Dear Eugen, We have a scenario of DC and DR replication, and planned to explore RBD mirroring with both Journaling and Snapshot mechanism. I have a 5 TB storage at Primary DC and 5 TB storage at DR site with 2 different ceph clusters configured. Please clarify the following queries 1. With One way mirroring, the failover works fine in both journaling and snapshot mechanism and we are able to promote the workload from DR site. How does Failback work? We wanted to move the contents from DR to DC but it fails. In journaling mechanism, it deletes the entire volume and recreates it afresh which does not solve our problem. 2. How does incremental replication work from DR to DC? 3. Does Two-way mirroring help this situation. According to me, in this method, it is for 2 different clouds with 2 different storages and replicating both the clouds workloads? Does Failback work in this scenario ? Please help us / guide us to deploy this solution Regards V.A.Prabha Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

3 4

Call for Proposals: Cephalocon 2024
by Noah Lehman 09 May '24

09 May '24

Hi Ceph community, We are excited to share the CFP for Cephalocon 2024 <https://events.linuxfoundation.org/cephalocon/program/cfp/>, running December 4-5 in Geneva, Switzerland. The deadline to submit a proposal is August 11, 2024. You can submit a proposal here <https://events.linuxfoundation.org/cephalocon/program/cfp/>. Running 4-5 December 2024, Cephalocon is the premier yearly event that brings together the global community of operators, developers, and researchers to celebrate Ceph, the open source distributed storage system designed to provide excellent performance, reliability, and scalability. Join new and existing community members from around the world at CERN to learn more about Ceph and the future of the project from the developers writing the code and the operators deploying it. *DATES TO REMEMBER:* CFP Opens: 6 May, 2024 CFP Close: 11 August, 2024 Speaker Notifications: 9 September, 2024 Schedule Announced: 12 September, 2024 Slides due date: 2 December, 2024 Event Dates: 4 December, 2024 - 5 December, 2024 Best, Noah

1 0

How to handle incomplete data after rbd import-diff failure?
by Satoru Takeuchi 09 May '24

09 May '24

I confirmed that incomplete data is left on `rbd import-diff` failure. I guess that this data is the part of snapshot. Could someone answer me the following questions? Q1. Is it safe to use the RBD image (e.g. client I/O and snapshot management) even though incomplete data exists? Q2. Is there any way to clean up the incomplete data? I read the following document and understand that this problem will be resolved after running `rbd import-diff` again. https://ceph.io/en/news/blog/2013/incremental-snapshots-with-rbd/ > Since overwriting the same data is idempotent, it’s safe to have an import-diff interrupted in the middle. However, it's difficult if I can't access the exported backup data anymore. For instance, I'm afraid of the following scenario. 1. Send the backup data from one DC (DC0) to another DC (DC1) periodically. 2. The backup data is created in DC0 and is sent directly to DC1 without persist backup data as a file. 3. Major power outage happens in DC0 and it's impossible to re-generate the backup data for a long time. I simulated this problem as follows: 1. Create an RBD image. 2. Write some data to this image. 3. Create a snapshot S0. 4. Write another data to this image. 5. Create a snapshot S1. 6. Create a backup data consists of the difference between S0 and S1 by running rbd export-diff. 7. Delete the last byte of the backup data, which is 'e' and means the end of the backup data, to inject import-diff failure. 8. Delete S1. 9. Run rbd import-diff to apply the broken backup data created in the step 7. Then step9 failed and S1 was not created. However, the number of RADOS objects and the storage usage has increased. before ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 11 MiB 24 9 24 0 0 0 3609 53 MiB 279 41 MiB 0 B 0 B total_objects 24 total_used 39 MiB total_avail 32 GiB total_space 32 GiB ``` after: ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 12 MiB 25 9 25 0 0 0 3531 53 MiB 278 41 MiB 0 B 0 B total_objects 25 total_used 40 MiB total_avail 32 GiB total_space 32 GiB ``` The incomplete data seem to increase if rbd import-diff fails again and again. The following output was get after the above-mentioned step9 100 times. ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 12 MiB 25 9 25 0 0 0 7925 104 MiB 1308 164 MiB 0 B 0 B total_objects 25 total_used 58 MiB total_avail 32 GiB total_space 32 GiB ``` Thanks, Satoru

2 3

Guidance on using large RBD volumes - NTFS
by Robert W. Eckert 08 May '24

08 May '24

Hi - in my home , I have been running cephfs for a few years, and have reasonably good performance, however since exposing cephfs via SMB has been hit and miss. So I thought I could carve out space for a RBD device to share from a windows machine My set up: CEPH 18.2.2 deployed using ceph adm 4 servers running RHEL9 on AMD 5600g CPUs 64 Gb Each 10Gbe NICs 4x4Tb Hdd 1 2Tb NVME for DB/WAL rbd pool is set to auto PG - its currently at 256 I have tested the NIC connection between the servers and my PC, an each point to point works well at the 10Gbe speeds Now the problem I created a 8Tb RBD using rbd create winshare -size 8T -pool rbd rbd map winshare I prepped the drive, formatted it, and the drive appears cleanly as an 8 Tb drive. When I used fio on the drive/volume, speeds were good around 150-200 Mb/s. Then I started trying to populate the drive from a few different sources, and performance took a nose dive. - Write speeds are about 6-10 Mb/s, and windows task manager shows average response time anywhere from 500ms to 30 seconds. - mainly around 4 seconds. I don't see any obvious bottlenecks - cpu on the servers are about 5-10%, memory is good, Network is showing under 1 Gb/s on all servers. I am wondering if I needed to use different parameters for creating the volume? Or is there a practical limit to the volume size I exceeded? Thanks, Rob

1 1

cephfs filesystem offline, ceph-mds core dumped... How to recover?
by Erich Weiler 08 May '24

08 May '24

Hi All, We just noticed that our cephfs filesystem was unreachable and we saw this: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf mds: 2/2 daemons up osd: 46 osds: 46 up (since 3d), 46 in (since 8w) data: volumes: 0/1 healthy, 1 recovering pools: 4 pools, 1313 pgs objects: 309.96M objects, 558 TiB usage: 842 TiB used, 287 TiB / 1.1 PiB avail pgs: 1293 active+clean 13 active+clean+scrubbing 7 active+clean+scrubbing+deep io: client: 1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr [root@pr-md-02 ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available [WRN] FS_DEGRADED: 1 filesystem is degraded fs slugfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more And filesystem status is: # ceph fs status slugfs - 31 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) slugfs.pr-md-02.sbblqq 0 0 0 0 1 active slugfs.pr-md-01.xdtppo Reqs: 0 /s 7958 7959 13.1k 442 POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) The ceph-mds container is not running on pr-md-02. All OSDs are online. I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there: # tail -1000 /var/log/messages May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May 7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167 May 7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May 7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. May 7 17:05:03 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May 7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May 7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May 7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May 7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May 7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167 May 7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May 7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. May 7 17:05:06 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "41": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "43": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "devices": [ May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ], May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_size": "1920378863616", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "tags": { May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: }, May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "type": "block", May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: ] May 7 17:05:06 pr-md-02 goofy_tesla[2564282]: } May 7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May 7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167 May 7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. May 7 17:05:07 pr-md-02 systemd[1]: Started libcrun container. May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 43, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_id": 41, May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "type": "bluestore" May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: } May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May 7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May 7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May 7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May 7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May 7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May 7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root. May 7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May 7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May 7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May 7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May 7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May 7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May 7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May 7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May 7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May 7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167 May 7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May 7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May 7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. May 7 17:06:11 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May 7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May 7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167 May 7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May 7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May 7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May 7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. May 7 17:06:12 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2) May 7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "41": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "43": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ], May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "tags": { May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: }, May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "type": "block", May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: ] May 7 17:06:13 pr-md-02 affectionate_saha[2565157]: } May 7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) May 7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May 7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. May 7 17:06:13 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167 May 7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May 7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May 7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May 7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. May 7 17:06:14 pr-md-02 systemd[1]: Started libcrun container. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May 7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May 7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 43, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_id": 41, May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "type": "bluestore" May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: } May 7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May 7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May 7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May 7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May 7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225254: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225255: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225256: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:26] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225257: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225258: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225259: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225260: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225261: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:36] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225262: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225263: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225264: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225265: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225266: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:46] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May 7 17:06:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225267: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225268: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225269: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225270: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:06:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225271: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:56] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:06:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225272: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:06:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:06:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225273: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228438. Immutable memtables: 0. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.708981) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12100] Flushing memtable with next log file: 228438 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819709068, "job": 12100, "event": "flush_started", "num_memtables": 1, "num_entries": 3587, "num_deletes": 921, "total_data_size": 2910092, "memory_usage": 2992096, "flush_reason": "Manual Compaction"} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12100] Level-0 flush table #228439: started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819721834, "cf_name": "default", "job": 12100, "event": "table_file_creation", "file_number": 228439, "file_size": 1847607, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141701983, "largest_seqno": 141705565, "table_properties": {"data_size": 1834448, "index_size": 6860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5253, "raw_key_size": 55412, "raw_average_key_size": 26, "raw_value_size": 1798552, "raw_average_value_size": 857, "num_data_blocks": 265, "num_entries": 2097, "num_filter_entries": 2097, "num_deletions": 921, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126677, "oldest_key_time": 1715126677, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228439, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12100] Flush lasted 12882 microseconds, and 5541 cpu microseconds. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721870) [db/flush_job.cc:967] [default] [JOB 12100] Level-0 flush table #228439: 1847607 bytes OK May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721889) [db/memtable_list.cc:519] [default] Level-0 commit table #228439 started May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724483) [db/memtable_list.cc:722] [default] Level-0 commit table #228439: memtable #1 done May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724523) EVENT_LOG_v1 {"time_micros": 1715126819724510, "job": 12100, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724561) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12100] Try to delete WAL files size 2890782, prev total WAL file size 2890782, number of live WAL files 2. May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725271) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337383930' seq:72057594037927935, type:22 .. '7061786F73003130313338313432' seq:0, type:0; will stop at (end) May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12101] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12100 Base level 0, inputs: [228439(1804KB)], [228437(36MB)] May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819725328, "job": 12101, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228439], "files_L6": [228437], "score": -1, "input_data_size": 39995951, "oldest_snapshot_seqno": -1} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12101] Generated table #228440: 25379 keys, 38251292 bytes, temperature: kUnknown May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819875843, "cf_name": "default", "job": 12101, "event": "table_file_creation", "file_number": 228440, "file_size": 38251292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 38096937, "index_size": 89812, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702456, "raw_average_key_size": 27, "raw_value_size": 37648320, "raw_average_value_size": 1483, "num_data_blocks": 3163, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228440, "seqno_to_time_mapping": "N/A"}} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.876095) [db/compaction/compaction_job.cc:1663] [default] [JOB 12101] Compacted 1@0 + 1@6 files to L6 => 38251292 bytes May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878629) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 265.6 rd, 254.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.8, 36.4 +0.0 blob) out(36.5 +0.0 blob), read-write-amplify(42.4) write-amplify(20.7) OK, records in: 27250, records dropped: 1871 output_compression: NoCompression May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878669) EVENT_LOG_v1 {"time_micros": 1715126819878658, "job": 12101, "event": "compaction_finished", "compaction_time_micros": 150577, "compaction_time_cpu_micros": 64114, "output_level": 6, "num_output_files": 1, "total_output_size": 38251292, "num_input_records": 27250, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819878937, "job": 12101, "event": "table_file_deletion", "file_number": 228439} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819882652, "job": 12101, "event": "table_file_deletion", "file_number": 228437} May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882737) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882741) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882743) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882744) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882746) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:07:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225274: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225275: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225276: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:06] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May 7 17:07:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225277: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:07:07 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_data', 'cephfs_md_and_data', '.mgr', 'cephfs_metadata'] May 7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:07:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:07:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225278: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225279: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225280: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:07:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225281: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 e5614: 46 total, 46 up, 46 in May 7 17:07:15 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225283: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d807ba79-465d-4d89-ab03-9bff6275dd10 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9e26e8f6-fba9-425c-923b-ba9da32693a7 does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1f435daa-e8f0-4964-b7fb-07ea14b95e9f does not exist May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.694558647 -0700 PDT m=+0.039788036 container create 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.722272344 -0700 PDT m=+0.067501679 container init 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May 7 17:07:17 pr-md-02 stoic_mayer[2565752]: 167 167 May 7 17:07:17 pr-md-02 systemd[1]: libpod-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728419093 -0700 PDT m=+0.073648421 container start 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728522035 -0700 PDT m=+0.073751360 container attach 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728769341 -0700 PDT m=+0.073998662 container died 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD) May 7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.678321769 -0700 PDT m=+0.023551090 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:17 pr-md-02 podman[2565757]: 2024-05-07 17:07:17.78259101 -0700 PDT m=+0.046708660 container remove 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:17 pr-md-02 systemd[1]: libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.90851263 -0700 PDT m=+0.039669496 container create 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May 7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope. May 7 17:07:17 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.936966849 -0700 PDT m=+0.068123675 container init 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943702066 -0700 PDT m=+0.074858891 container start 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943794994 -0700 PDT m=+0.074951827 container attach 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.892929482 -0700 PDT m=+0.024086307 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> passed data devices: 2 physical, 0 LVM May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> relative data size: 1.0 May 7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> All data devices are unavailable May 7 17:07:18 pr-md-02 systemd[1]: libpod-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 podman[2565778]: 2024-05-07 17:07:18.556739188 -0700 PDT m=+0.687896012 container died 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:18 pr-md-02 podman[2565822]: 2024-05-07 17:07:18.614711286 -0700 PDT m=+0.047414226 container remove 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May 7 17:07:18 pr-md-02 systemd[1]: libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May 7 17:07:18 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-2f63fd9ffd2763a7ee3f75befaf99b27a75325cd025cc2e74e72d7bd150afc42-merged.mount: Deactivated successfully. May 7 17:07:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225284: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.171860525 -0700 PDT m=+0.039261474 container create eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.199519439 -0700 PDT m=+0.066920407 container init eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:19 pr-md-02 xenodochial_wu[2565974]: 167 167 May 7 17:07:19 pr-md-02 systemd[1]: libpod-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.205924433 -0700 PDT m=+0.073325392 container start eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.2060646 -0700 PDT m=+0.073465559 container attach eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.206298065 -0700 PDT m=+0.073699031 container died eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May 7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.158156321 -0700 PDT m=+0.025557273 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 podman[2565979]: 2024-05-07 17:07:19.25994464 -0700 PDT m=+0.046894141 container remove eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.388083595 -0700 PDT m=+0.043236647 container create a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) May 7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope. May 7 17:07:19 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.417053374 -0700 PDT m=+0.072206356 container init a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423715752 -0700 PDT m=+0.078868733 container start a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423810643 -0700 PDT m=+0.078963634 container attach a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.371099332 -0700 PDT m=+0.026252317 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-91f5899d00d5794e67bbdd45fa72d22c74b0b977efec0238aa842ac272b635ce-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "41": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme0n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "41", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "43": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "devices": [ May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme1n1" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ], May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_size": "1920378863616", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "tags": { May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "43", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: }, May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "type": "block", May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: ] May 7 17:07:19 pr-md-02 agitated_mayer[2566015]: } May 7 17:07:19 pr-md-02 systemd[1]: libpod-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.801695942 -0700 PDT m=+0.456848932 container died a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-68bc380ea9ec1cff282d08f862c7b661ba0571a20622225a5a3994e5fc88985d-merged.mount: Deactivated successfully. May 7 17:07:19 pr-md-02 podman[2566028]: 2024-05-07 17:07:19.854983202 -0700 PDT m=+0.045986655 container remove a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May 7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.420915201 -0700 PDT m=+0.041065865 container create f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.45143613 -0700 PDT m=+0.071586769 container init f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May 7 17:07:20 pr-md-02 practical_wu[2566180]: 167 167 May 7 17:07:20 pr-md-02 systemd[1]: libpod-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457513493 -0700 PDT m=+0.077664122 container start f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457638676 -0700 PDT m=+0.077789308 container attach f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457913249 -0700 PDT m=+0.078063874 container died f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May 7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.405495878 -0700 PDT m=+0.025646501 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 podman[2566185]: 2024-05-07 17:07:20.515553442 -0700 PDT m=+0.045837844 container remove f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) May 7 17:07:20 pr-md-02 systemd[1]: libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.631476902 -0700 PDT m=+0.043759539 container create c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) May 7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope. May 7 17:07:20 pr-md-02 systemd[1]: Started libcrun container. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.660235834 -0700 PDT m=+0.072518401 container init c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.666915701 -0700 PDT m=+0.079198260 container start c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD) May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.667002215 -0700 PDT m=+0.079284787 container attach c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) May 7 17:07:20 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb2679257a24186947b435237af458658d281389b531640688ba39cfa7cac79-merged.mount: Deactivated successfully. May 7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.6134279 -0700 PDT m=+0.025710470 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May 7 17:07:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225285: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 43, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: }, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_id": 41, May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: "type": "bluestore" May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 quirky_clarke[2566222]: } May 7 17:07:21 pr-md-02 systemd[1]: libpod-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566207]: 2024-05-07 17:07:21.655826716 -0700 PDT m=+1.068109291 container died c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May 7 17:07:21 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-e573f9cf164334d9b913d82ba760fa7f00e98646ea067755c99a3842969d30d8-merged.mount: Deactivated successfully. May 7 17:07:21 pr-md-02 podman[2566303]: 2024-05-07 17:07:21.711419562 -0700 PDT m=+0.047850993 container remove c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1) May 7 17:07:21 pr-md-02 systemd[1]: libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 57f8febe-215e-403c-821d-95da6e3c5073 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 2637522f-3bd6-453c-976c-f7da78d12399 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b0cc7927-d150-4870-8ec7-6ef7d847ffd2 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9b053170-587d-42ee-929e-25112b3abba1 does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 352cfbfb-4b0a-420c-959b-e42615f9913d does not exist May 7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 144dbb62-1eeb-49ba-b928-eee4a1756251 does not exist May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May 7 17:07:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225286: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225287: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225288: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:29 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225289: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:31 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225290: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:33 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225291: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:35 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225292: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May 7 17:07:37 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225293: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:39 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225294: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:41 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225295: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:43 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225296: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:45 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225297: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:46] "GET /metrics HTTP/1.1" 200 73538 "" "Prometheus/2.43.0" May 7 17:07:47 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225298: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:49 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225299: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:51 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225300: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:53 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225301: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May 7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May 7 17:07:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225302: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:56] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:07:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225303: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:07:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:07:59 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:07:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225304: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225305: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225306: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:04 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225307: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub starts May 7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub ok May 7 17:08:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:06] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May 7 17:08:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225308: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 e5615: 46 total, 46 up, 46 in May 7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:08:07 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', 'cephfs_data', '.mgr', 'cephfs_metadata'] May 7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May 7 17:08:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May 7 17:08:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225310: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:09 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May 7 17:08:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225311: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225312: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automate… May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May 7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May 7 17:08:14 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May 7 17:08:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225313: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:16] "GET /metrics HTTP/1.1" 200 73536 "" "Prometheus/2.43.0" May 7 17:08:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225314: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May 7 17:08:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02(a)1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228441. Immutable memtables: 0. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.105603) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12102] Flushing memtable with next log file: 228441 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898105644, "job": 12102, "event": "flush_started", "num_memtables": 1, "num_entries": 2046, "num_deletes": 617, "total_data_size": 1704550, "memory_usage": 1744008, "flush_reason": "Manual Compaction"} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12102] Level-0 flush table #228442: started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898113765, "cf_name": "default", "job": 12102, "event": "table_file_creation", "file_number": 228442, "file_size": 847046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141705570, "largest_seqno": 141707611, "table_properties": {"data_size": 839933, "index_size": 2991, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 32187, "raw_average_key_size": 26, "raw_value_size": 819966, "raw_average_value_size": 670, "num_data_blocks": 115, "num_entries": 1223, "num_filter_entries": 1223, "num_deletions": 617, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126820, "oldest_key_time": 1715126820, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228442, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12102] Flush lasted 8207 microseconds, and 2745 cpu microseconds. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113810) [db/flush_job.cc:967] [default] [JOB 12102] Level-0 flush table #228442: 847046 bytes OK May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113830) [db/memtable_list.cc:519] [default] Level-0 commit table #228442 started May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116583) [db/memtable_list.cc:722] [default] Level-0 commit table #228442: memtable #1 done May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116634) EVENT_LOG_v1 {"time_micros": 1715126898116612, "job": 12102, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116650) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12102] Try to delete WAL files size 1693223, prev total WAL file size 1693550, number of live WAL files 2. May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228438.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.117035) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740032393030383037' seq:72057594037927935, type:22 .. '6D6772737461740032393031303630' seq:0, type:0; will stop at (end) May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12103] Compacting 1@0 + 1@6 files to L6, score -1.00 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12102 Base level 0, inputs: [228442(827KB)], [228440(36MB)] May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898117062, "job": 12103, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228442], "files_L6": [228440], "score": -1, "input_data_size": 39098338, "oldest_snapshot_seqno": -1} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12103] Generated table #228443: 25379 keys, 37258556 bytes, temperature: kUnknown May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898320688, "cf_name": "default", "job": 12103, "event": "table_file_creation", "file_number": 228443, "file_size": 37258556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37107602, "index_size": 86411, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702042, "raw_average_key_size": 27, "raw_value_size": 36662378, "raw_average_value_size": 1444, "num_data_blocks": 3038, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228443, "seqno_to_time_mapping": "N/A"}} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.321010) [db/compaction/compaction_job.cc:1663] [default] [JOB 12103] Compacted 1@0 + 1@6 files to L6 => 37258556 bytes May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323275) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 191.9 rd, 182.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.8, 36.5 +0.0 blob) out(35.5 +0.0 blob), read-write-amplify(90.1) write-amplify(44.0) OK, records in: 26602, records dropped: 1223 output_compression: NoCompression May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323329) EVENT_LOG_v1 {"time_micros": 1715126898323308, "job": 12103, "event": "compaction_finished", "compaction_time_micros": 203764, "compaction_time_cpu_micros": 65018, "output_level": 6, "num_output_files": 1, "total_output_size": 37258556, "num_input_records": 26602, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898323660, "job": 12103, "event": "table_file_deletion", "file_number": 228442} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898326958, "job": 12103, "event": "table_file_deletion", "file_number": 228440} May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116998) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326990) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326994) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327002) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327006) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May 7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327008) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting I have not updated any packages or anything, I'm not sure how this came about. I did set the following two items earlier today, although I'm not sure if it would make any difference? ceph config set mds mds_recall_max_caps 3000 ceph config set mds mds_cache_memory_limit 16000000000 I tried to restart the MDS daemons alike so: ceph orch daemon restart mds.slugfs.pr-md-02.sbblqq And they seem to start for a moment then stop again. I then tried to restart the one remaining one that was online, then it seemed to die as well, and now I see this: # ceph fs status slugfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 0 0 0 0 1 failed POOL TYPE USED AVAIL cephfs_metadata metadata 144G 3168G cephfs_md_and_data data 0 65.4T cephfs_data data 838T 130T MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) But the ceph-mds on pr-md-01 doesn't exist now. Anyone know how to recover from this? I'm kind of panicking. :( Thanks much, erich

3 6

How to define a read-only sub-user?
by Matthew Darwin 08 May '24

08 May '24

Hi, I'm new to bucket policies. I'm trying to create a sub-user that has only read-only access to all the buckets of the main user. I created the below policy, I can't create or delete files, but I can still create buckets using "rclone mkdir". Any idea what I'm doing wrong? I'm using ceph quincy. radosgw-admin subuser create --uid=main_user --subuser=eosn_read --access=read radosgw-admin key create --subuser=main_user:sub_user --key-type=s3 --access-key xxxx --secret-key xxxx s3cmd setpolicy policy.txt s3://somebucket { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/main_user:sub_user" ] }, "Action": [ "s3:ListBucket", "s3:ListAllMyBuckets", "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectTagging", "s3:GetObjectRetention", "s3:GetObjectLegalHold" ], "Resource": "arn:aws:s3:::*" }

1 0

Removed host in maintenance mode
by Johan 07 May '24

07 May '24

Hi all, In my small cluster of 6 hosts I had troubles with a host (osd:s) and was planning to remove it from the cluster. Before I got to do that I needed to power down this host and therefore put it in maintenance mode. Due to some mistakes on my part I couldn't boot the host again and simply decided to force the removal from the cluster. The host is now removed but ceph (17.2.7) keep on complaining on it being in maintenance mode. How can I remove the last remnants of this host from the cluster? /Johan

2 3

cephadm upgrade: heartbeat failures not considered
by Eugen Block 07 May '24

07 May '24

Hi, we're facing an issue during upgrades (and sometimes server reboots), it appears to occur when (at leat) one of the MONs has to do a full sync. And I'm wondering if the upgrade procedure could be improved in that regard, I'll come back to that later. First, I'll try to summarize the events. We upgraded to latest Pacific (16.2.15) last week. Note that the filesystems of the MONs are on HDDs (we're planning to move to flash, we already noticed issues during disaster recovery on HDDs last year). According to the logs the MGRs and MONs were upgraded successfully and a quorum was formed successfully at: 2024-04-30T12:57:22.347182+0000 mon.ndeceph03 (mon.0) 1208046 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum ndeceph03,ndeceph01) Then ceph started the first OSD upgrades a few minutes later, the first one at: 2024-04-30T13:00:42.733528+0000 mon.ndeceph03 (mon.0) 101 : cluster [INF] osd.25 marked itself down and dead One OSD by one on ndeceph01 was upgraded: 2024-04-30T13:01:46.517+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.19 (5/7) 2024-04-30T13:01:46.517+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.19 (5/7) 2024-04-30T13:02:24.800+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.22 (6/7) 2024-04-30T13:02:24.800+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.22 (6/7) 2024-04-30T13:02:48.220+0000 7fbd78a31700 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.29 (7/7) 2024-04-30T13:02:48.220+0000 7fbd78a31700 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.29 (7/7) But the mon service on ndeceph02 (also OSD server) was still syncing (for around 6 minutes): 2024-04-30T13:02:33.124+0000 7f1c24444700 1 mon.ndeceph02@2(synchronizing) e58 handle_auth_request failed to assign global_id 2024-04-30T13:08:30.123+0000 7f1c24444700 1 mon.ndeceph02@2(synchronizing) e58 handle_auth_request failed to assign global_id All HDD OSDs from ndeceph02 (not upgraded yet) were complaining about timeouts, and apparently tried to reboot multiple times (wondering why the SSDs didn't complain though): 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f2e1399b700' had timed out after 15.000000954s 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 osd.0 367411 is_healthy false -- internal heartbeat failed 2024-04-30T13:03:54.374+0000 7f2e2d1ce700 1 osd.0 367411 not healthy; waiting to boot 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f2e1399b700' had timed out after 15.000000954s 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 osd.0 367411 is_healthy false -- internal heartbeat failed 2024-04-30T13:03:55.330+0000 7f2e2d1ce700 1 osd.0 367411 not healthy; waiting to boot ... 2024-04-30T13:03:59.322+0000 7f2e201b4700 1 osd.0 367414 state: booting -> active 2024-04-30T13:05:22.184+0000 7f2e201b4700 1 osd.0 367433 state: booting -> active 2024-04-30T13:06:39.602+0000 7f2e201b4700 1 osd.0 367447 state: booting -> active 2024-04-30T13:08:02.612+0000 7f2e201b4700 1 osd.0 367454 state: booting -> active During this syncing period the disk utilization of the OS filesystem was at 100%, but ceph kept upgrading other OSD daemons. There are 3 main hosts, failure domain is host, replicated pools with min_size 2, size 3. If the OSDs from one host are struggling (because of disk IO on the filesystem, apparently) and ceph keeps upgrading others, we get inactive PGs. So what I'm wondering about is, how does the orchestrator decide if it's ok to stop one OSD ('ceph osd ok-to-stop <ID>' is the manual command) while others obviously are not healthy and would cause inactive PGs? The cluster did notice slow requests and reported messages like these: 2024-04-30T13:02:58.867499+0000 osd.28 (osd.28) 10529 : cluster [WRN] Monitor daemon marked osd.28 down, but it is still running If some OSDs clearly aren't healthy, I would expect the orchestrator to pause the upgrade. After the mon full sync completed, it started upgrading the OSDs on ndeceph02 as well, but the inactive PGs were only resolved after several more minutes when almost all OSDs from that tree branch had been upgraded: 2024-04-30T13:16:17.536359+0000 mon.ndeceph03 (mon.0) 1461 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 11 pgs peering) During a planned upgrade this issue can be mitigated via staggered upgrades (now that I know what the cause is), upgrading MGRs and MONs first and wait until everything has settled. Then continue with OSDs. But after a reboot there's no way to control that, of course. I helped a customer last year with a mon sync issue so we might be able to improve things a bit until we have flash disks. I don't know how it all behaves in newer ceph versions regarding the upgrade, but I feel like it wouldn't make a difference right now. Is there any way to improve the orchestrator to consider unhealthy OSDs before stopping healthy ones? Thanks! Eugen

1 0

cache pressure?
by Erich Weiler 07 May '24

07 May '24

So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure". Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time? I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...? We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping... -erich

3 9

Dashboard issue slowing to a crawl - active ceph mgr process spiking to 600%+
by Zachary Perry 07 May '24

07 May '24

Hello All, I'm hoping I can get some help with an issue in the dashboard after doing a recent bare metal ceph upgrade from Octopus to Quincy. ** Please note, this document references it only being an issue with the images tab shortly after this I found the same issue on another cluster that was recently upgraded from Octopus to quincy 17.2.7 within the last few months and it's affecting all tabs in the ceph dashboard it slows to a crawl until I restart or fail over the mgr both running on top of ubuntu 20.04 Everything appears to be working fine besides the Block --> images tab. It doesn't matter what node I fail over to, reboots, reinstalling ceph-mgr-dashboard, different broswers, clients etc It will not load the 4 RBDs I have, they appear in rbd ls, I can query them and the connection on the end appliance is fine. The loading icons spin infinitely without any failure message. If I access the images tab and then move to any other tab is the dashboard it will allow me to navigate but not display anything until I either restart the service on the active mgr or fail over to another, so it works as expected until I access this one tab. when I use any other section in the in the dashboard cpu utilization for the ceph-mgr is normal but when I access the images tab it's spiked to as high as 600% and will stay like that until I restart the service or fail over the active mgr -- Active MGR before clicking Block, the OSDs spike for a second but revert to around 5% top - 13:43:37 up 8 days, 23:09, 1 user, load average: 8.08, 5.02, 4.37 Tasks: 695 total, 1 running, 694 sleeping, 0 stopped, 0 zombie %Cpu(s): 7.0 us, 1.6 sy, 0.0 ni, 89.7 id, 1.2 wa, 0.0 hi, 0.5 si, 0.0 st ----- MiB Mem : 128474.1 total, 6705.6 free, 65684.0 used, 56084.5 buff/cache MiB Swap: 40927.0 total, 35839.3 free, 5087.7 used. 49253.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14156 ceph 20 0 3420632 1.9g 13668 S 55.3 1.5 864:49.51 ceph-osd 13762 ceph 20 0 3421384 1.8g 13432 S 51.3 1.4 960:22.12 ceph-osd 14163 ceph 20 0 3422352 1.7g 13016 S 50.0 1.3 902:41.19 ceph-osd 13803 ceph 20 0 3469596 1.8g 13532 S 44.7 1.4 941:55.10 ceph-osd 13774 ceph 20 0 3427560 1.7g 13656 S 38.7 1.4 932:02.51 ceph-osd 13801 ceph 20 0 3439796 1.7g 13448 S 37.7 1.3 981:25.55 ceph-osd 14025 ceph 20 0 3426360 1.8g 13780 S 36.4 1.4 994:00.75 ceph-osd 9888 nobody 20 0 126100 8696 0 S 21.2 0.0 1106:19 node_exporter 126798 ceph 20 0 1787824 528000 39464 S 7.9 0.4 0:14.84 ceph-mgr 13795 ceph 20 0 3420252 1.7g 13264 S 7.6 1.4 990:00.61 ceph-osd 13781 ceph 20 0 3484476 1.9g 13248 S 6.3 1.5 1040:10 ceph-osd 13777 ceph 20 0 3408972 1.8g 13464 S 6.0 1.5 1026:21 ceph-osd 13797 ceph 20 0 3432068 1.6g 13932 S 6.0 1.3 950:39.35 ceph-osd 13779 ceph 20 0 3471668 1.7g 12728 S 5.6 1.3 984:53.80 ceph-osd 13768 ceph 20 0 3496064 1.9g 13504 S 5.3 1.5 918:37.48 ceph-osd 13786 ceph 20 0 3422044 1.6g 13456 S 5.3 1.3 974:29.08 ceph-osd 13788 ceph 20 0 3454184 1.9g 13048 S 5.3 1.5 980:35.78 ceph-osd 13776 ceph 20 0 3445680 1.7g 12880 S 5.0 1.3 998:30.58 ceph-osd 13785 ceph 20 0 3409548 1.7g 13704 S 5.0 1.3 939:37.08 ceph-osd 14152 ceph 20 0 3465284 1.7g 13840 S 5.0 1.4 959:39.42 ceph-osd 10339 nobody 20 0 6256048 531428 60188 S 4.6 0.4 239:37.56 prometheus 13802 ceph 20 0 3430696 1.8g 13872 S 4.6 1.4 924:15.74 ceph-osd 13791 ceph 20 0 3498876 1.5g 12648 S 4.3 1.2 962:58.37 ceph-osd 13800 ceph 20 0 3455268 1.7g 12404 S 4.3 1.3 1000:41 ceph-osd 13790 ceph 20 0 3434364 1.6g 13516 S 3.3 1.3 974:16.46 ceph-osd 14217 ceph 20 0 3443436 1.8g 13560 S 3.3 1.4 902:54.22 ceph-osd 13526 ceph 20 0 1012048 499628 11244 S 3.0 0.4 349:35.28 ceph-mon 13775 ceph 20 0 3367284 1.6g 13940 S 3.0 1.3 878:38.27 ceph-osd 13784 ceph 20 0 3380960 1.8g 12892 S 3.0 1.4 910:50.47 ceph-osd 13789 ceph 20 0 3432876 1.6g 12464 S 2.6 1.2 922:45.15 ceph-osd 13804 ceph 20 0 3428120 1.9g 13192 S 2.6 1.5 865:31.30 ceph-osd 14153 ceph 20 0 3432752 1.8g 12576 S 2.3 1.4 874:27.92 ceph-osd 14192 ceph 20 0 3412640 1.9g 13512 S 2.3 1.5 923:01.97 ceph-osd 13796 ceph 20 0 3433016 1.8g 13164 S 2.0 1.4 982:08.21 ceph-osd 13798 ceph 20 0 3405708 1.6g 13508 S 2.0 1.3 873:50.34 ceph-osd 13814 ceph 20 0 4243252 1.5g 13500 S 2.0 1.2 2020:41 ceph-osd 13985 ceph 20 0 3487848 1.6g 13100 S 2.0 1.3 942:21.96 ceph-osd 14001 ceph 20 0 4194336 1.9g 13460 S 2.0 1.5 2143:46 ceph-osd 14186 ceph 20 0 3441852 1.5g 13360 S 2.0 1.2 956:30.81 ceph-osd 7257 root 20 0 82332 3480 2984 S 0.3 0.0 9:22.50 irqbalance 7269 syslog 20 0 224344 3648 2392 S 0.3 0.0 0:11.79 rsyslogd 16621 472 20 0 898376 79800 11080 S 0.3 0.1 146:36.08 grafana 104366 root 20 0 0 0 0 I 0.3 0.0 1:11.30 kworker/0:2- events 125676 root 20 0 0 0 0 I 0.3 0.0 0:00.48 kworker/u104:7- public-bond 127115 root 20 0 10172 4636 3392 R 0.3 0.0 0:00.10 top 1 root 20 0 180712 16312 5652 S 0.0 0.0 1:31.85 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.15 kthreadd --- top - 13:44:28 up 8 days, 23:09, 1 user, load average: 6.79, 5.11, 4.43 Tasks: 695 total, 1 running, 694 sleeping, 0 stopped, 0 zombie %Cpu(s): 12.9 us, 2.1 sy, 0.0 ni, 83.5 id, 1.0 wa, 0.0 hi, 0.5 si, 0.0 st MiB Mem : 128474.1 total, 6219.8 free, 66504.7 used, 55749.6 buff/cache MiB Swap: 40927.0 total, 35837.0 free, 5090.0 used. 48435.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 126798 ceph 20 0 1854596 573432 41484 S 482.5 0.4 1:45.70 ceph-mgr 14156 ceph 20 0 3420632 1.9g 13668 S 54.6 1.5 865:16.58 ceph-osd 13762 ceph 20 0 3421384 1.9g 13432 S 51.7 1.5 960:47.47 ceph-osd 13803 ceph 20 0 3469596 1.9g 13532 S 49.3 1.5 942:18.23 ceph-osd 14163 ceph 20 0 3422352 1.9g 13016 S 49.3 1.5 903:05.84 ceph-osd 13795 ceph 20 0 3420252 1.8g 13264 S 7.0 1.4 990:04.41 ceph-osd 13777 ceph 20 0 3408972 1.9g 13464 S 5.6 1.5 1026:24 ceph-osd 13786 ceph 20 0 3422044 1.6g 13456 S 5.3 1.3 974:32.09 ceph-osd 13797 ceph 20 0 3432068 1.6g 13932 S 5.3 1.3 950:42.02 ceph-osd 16621 472 20 0 898376 78776 11044 S 5.3 0.1 146:36.76 grafana 13791 ceph 20 0 3498876 1.5g 12648 S 5.0 1.2 963:00.94 ceph-osd 14001 ceph 20 0 4194336 1.9g 13460 S 5.0 1.5 2143:47 ceph-osd 9888 nobody 20 0 126100 8696 0 S 4.6 0.0 1106:23 node_exporter 13768 ceph 20 0 3496064 1.9g 13504 S 4.6 1.5 918:39.85 ceph-osd 13776 ceph 20 0 3445680 1.7g 12880 S 4.6 1.3 998:32.98 ceph-osd 13781 ceph 20 0 3484476 1.6g 13248 S 4.6 1.3 1040:13 ceph-osd 13785 ceph 20 0 3409548 1.7g 13704 S 4.6 1.3 939:40.08 ceph-osd 13788 ceph 20 0 3454184 1.9g 13048 S 4.6 1.5 980:38.24 ceph-osd 13779 ceph 20 0 3471668 1.7g 12728 S 4.3 1.3 984:56.39 ceph-osd 13800 ceph 20 0 3455268 1.7g 12404 S 4.3 1.3 1000:44 ceph-osd 13802 ceph 20 0 3430696 1.8g 13872 S 4.0 1.4 924:18.14 ceph-osd 14152 ceph 20 0 3465284 1.7g 13840 S 4.0 1.4 959:41.83 ceph-osd 13796 ceph 20 0 3433016 1.8g 13164 S 3.0 1.4 982:09.66 ceph-osd 13784 ceph 20 0 3380960 1.8g 12892 S 2.6 1.4 910:52.06 ceph-osd 13790 ceph 20 0 3434364 1.6g 13516 S 2.6 1.3 974:17.99 ceph-osd 13801 ceph 20 0 3439796 1.9g 13448 S 2.6 1.5 981:42.61 ceph-osd 14153 ceph 20 0 3432752 1.8g 12576 S 2.6 1.4 874:29.30 ceph-osd 14186 ceph 20 0 3441852 1.6g 13360 S 2.6 1.2 956:32.32 ceph-osd 14192 ceph 20 0 3412640 1.9g 13512 S 2.6 1.5 923:03.59 ceph-osd 13526 ceph 20 0 1012048 496332 11208 S 2.3 0.4 349:36.89 ceph-mon 13789 ceph 20 0 3432876 1.6g 12464 S 2.3 1.2 922:46.59 ceph-osd 13798 ceph 20 0 3405708 1.6g 13508 S 2.3 1.3 873:51.73 ceph-osd 14217 ceph 20 0 3443436 1.8g 13560 S 2.3 1.4 902:55.74 ceph-osd 13774 ceph 20 0 3427560 1.9g 13656 S 2.0 1.5 932:19.26 ceph-osd 13775 ceph 20 0 3367284 1.6g 13940 S 2.0 1.3 878:39.76 ceph-osd 13814 ceph 20 0 4243252 1.5g 13500 S 2.0 1.2 2020:42 ceph-osd 13985 ceph 20 0 3487848 1.6g 13100 S 2.0 1.3 942:23.39 ceph-osd 13804 ceph 20 0 3428120 1.9g 13192 S 1.7 1.5 865:32.71 ceph-osd 14025 ceph 20 0 3426360 1.8g 13780 S 1.7 1.5 994:17.32 ceph-osd 10339 nobody 20 0 6256048 537184 60136 S 1.0 0.4 239:38.44 prometheus 17547 nobody 20 0 128448 8572 0 S 1.0 0.0 31:54.99 alertmanager 127115 root 20 0 10172 4636 3392 R 0.7 0.0 0:00.43 top --- OS: Ubuntu 20.04 128GB of memory Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz X11SPL-F --- Ceph Versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 140 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "ctdb": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 1 }, "rgw": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 154 } } --- cluster: id: 388dda42-9dd0-4858-a978-b3dc4c3b9152 health: HEALTH_OK services: mon: 3 daemons, quorum jarn29,jarn30,jarn31 (age 8d) mgr: osd31(active, since 10m), standbys: osd30, osd29, osd32 mds: 2/2 daemons up, 2 standby osd: 140 osds: 140 up (since 3d), 140 in (since 3d) flags noautoscale ctdb: 1 daemon active (1 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 19 pools, 5024 pgs objects: 792.78M objects, 865 TiB usage: 1.4 PiB used, 586 TiB / 2.0 PiB avail pgs: 5013 active+clean 11 active+clean+scrubbing+deep io: client: 19 MiB/s rd, 102 MiB/s wr, 672 op/s rd, 351 op/s wr --- I've ran perf and included the ceph-mgr.log for the first system that is displaying the issue Perf - https://imgur.com/a/VMh4tDf mgr log while accessing RBD tab - https://pastebin.com/t96WCWfc mgr logs prior to clicking RBD -https://pastebin.com/e4dtuD3i --- Apologies for the formatting first time posting here If anything else is needed please let me know!

2 1

Mysterious Space-Eating Monster
by duluxoz 07 May '24

07 May '24

Hi All, *Something* is chewing up a lot of space on our `\var` partition to the point where we're getting warnings about the Ceph monitor running out of space (ie > 70% full). I've been looking, but I can't find anything significant (ie log files aren't too big, etc) BUT there seem to be a hell of a lot (15) of sub-directories (with GUIDs for names) under the `/var/lib/containers/storage/overlay/` folder, all ending with `merged` - ie `/var/lib/containers/storage/overlay/{{GUID}}/`merged`. Is this normal, or is something going wrong somewhere, or am I looking in the wrong place? Also, if this is the issue, can I delete these folders? Sorry for asking such a noob Q, but the Cephadm/Podman stuff is extremely new to me :-) Thanks in advance Cheers Dulux-Oz

3 3

MDS 17.2.7 crashes at rejoin
by Robert Sander 07 May '24

07 May '24

Hi, a 17.2.7 cluster with two filesystems has suddenly non-working MDSs: # ceph -s cluster: id: f54eea86-265a-11eb-a5d0-457857ba5742 health: HEALTH_ERR 22 failed cephadm daemon(s) 2 filesystems are degraded 1 mds daemon damaged insufficient standby MDS daemons available services: mon: 5 daemons, quorum ceph00,ceph03,ceph04,ceph01,ceph02 (age 4h) mgr: ceph03.odfupq(active, since 4h), standbys: ppc721.vsincn, ceph00.lvbddp, ceph02.zhyxjg, ceph06.eifppc mds: 4/5 daemons up osd: 145 osds: 145 up (since 20h), 145 in (since 2d) rgw: 12 daemons active (4 hosts, 1 zones) data: volumes: 0/2 healthy, 2 recovering; 1 damaged pools: 15 pools, 4897 pgs objects: 195.64M objects, 195 TiB usage: 617 TiB used, 527 TiB / 1.1 PiB avail pgs: 4892 active+clean 5 active+clean+scrubbing+deep io: client: 2.5 MiB/s rd, 20 MiB/s wr, 665 op/s rd, 938 op/s wr # ceph fs status ABC - 4 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 resolve ABC.ceph04.lzlkdu 0 3 1 0 2 resolve ABC.ppc721.rzfmyi 0 3 1 0 3 resolve ABC.ceph04.jiepaw 249 252 13 0 POOL TYPE USED AVAIL cephfs.ABC.meta metadata 33.0G 104T cephfs.ABC.data data 390T 104T DEF - 154 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 rejoin(laggy) DEF.ceph06.etthum 30.9k 30.8k 5084 0 POOL TYPE USED AVAIL cephfs.DEF.meta metadata 190G 104T cephfs.DEF.data data 118T 104T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) The first filesystem will not get an MDS in rank 0, we already tried to set max_msd to 1 but to no avail. The second filesystem's MDS shows "replay" for a while and then it crashes in the rejoin phase with: -92> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 handle_mds_map i am now mds.0.501522 -91> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 handle_mds_map state change up:reconnect --> up:rejoin -90> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 rejoin_start -89> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 rejoin_joint_start -88> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfece err -22/0 -87> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -86> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -85> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -84> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -83> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -82> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -81> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ebd err -22/-22 -80> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ecd err -22/-22 -79> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9ea err -22/-22 -78> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -77> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9c3 err -22/-22 -76> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc978 err -22/-22 -75> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -74> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -73> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -72> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -71> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7e err -22/-22 -70> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -69> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -68> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -67> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -66> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ebd err -22/-22 -65> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ecd err -22/-22 -64> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9ea err -22/-22 -63> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc978 err -22/-22 -62> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x3000069373a err -22/-22 -61> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012dc5d8 err -22/-22 -60> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a32e8 err -22/-22 -59> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000696952 err -22/-22 -58> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -57> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -56> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -55> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -54> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7e err -22/-22 -53> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -52> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012c3a0e err -22/-22 -51> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -50> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -49> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7f err -22/-22 -48> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -47> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -46> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -45> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -44> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac2 err -22/-22 -43> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -42> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -41> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -40> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -39> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012c3a0e err -22/-22 -38> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -37> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x40000d63bff err -22/-22 -36> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -35> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -34> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -33> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac2 err -22/-22 -32> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -31> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac4 err -22/-22 -30> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -29> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -28> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -27> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -26> 2024-05-06T16:07:15.530+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -25> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -24> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -23> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -22> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -21> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a5fc4 err -22/-22 -20> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a634d err -22/-22 -19> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a63bf err -22/-22 -18> 2024-05-06T16:07:15.542+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -17> 2024-05-06T16:07:15.542+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -16> 2024-05-06T16:07:15.546+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -15> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -14> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -13> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a5fc4 err -22/-22 -12> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a634d err -22/-22 -11> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a63bf err -22/-22 -10> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bfd9c err -22/-22 -9> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bfb78 err -22/-22 -8> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -7> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -6> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -5> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -4> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -3> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -2> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x40000d5a226 err -22/-22 -1> 2024-05-06T16:07:15.634+0000 7f1921e91700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: In function 'void MDCache::rejoin_send_rejoins()' thread 7f1921e91700 time 2024-05-06T16:07:15.635683+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: 4086: FAILED ceph_assert(auth >= 0) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1930ad94a3] 2: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f1930ad9669] 3: (MDCache::rejoin_send_rejoins()+0x216b) [0x5614ac8747eb] 4: (MDCache::process_imported_caps()+0x1993) [0x5614ac872353] 5: (Context::complete(int)+0xd) [0x5614ac6e182d] 6: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 7: (void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x8d) [0x5614ac6e6f5d] 8: (OpenFileTable::_open_ino_finish(inodeno_t, int)+0x156) [0x5614aca765a6] 9: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 10: (void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x8d) [0x5614ac6e6f5d] 11: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0x138) [0x5614ac867168] 12: (MDCache::_open_ino_backtrace_fetched(inodeno_t, ceph::buffer::v15_2_0::list&, int)+0x290) [0x5614ac87ff90] 13: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 14: (MDSIOContextBase::complete(int)+0x534) [0x5614aca426e4] 15: (Finisher::finisher_thread_entry()+0x18d) [0x7f1930b7884d] 16: /lib64/libpthread.so.0(+0x81ca) [0x7f192fac81ca] 17: clone() How do we solve this issue? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 3

MDS crashes shortly after starting
by E Taka 07 May '24

07 May '24

Hi all, we have a serious problem with CephFS. A few days ago, the CephFS file systems became inaccessible, with the message MDS_DAMAGE: 1 mds daemon damaged The cephfs-journal-tool tells us: "Overall journal integrity: OK" The usual attempts with redeploy were unfortunately not successful. After many attempts to achieve something with the orchestrator, we set the MDS to “failed” and provoked the creation of new MDS with “ceph fs reset”. But this MDS crashes: ceph-17.2.7/src/mds/MDCache.cc: In function 'void MDCache::rejoin_send_rejoins()' ceph-17.2.7/src/mds/MDCache.cc: 4086: FAILED ceph_assert(auth >= 0) (The full trace is attached). What can we do now? We are grateful for any help!

2 1

Luminous OSDs failing with FAILED assert(clone_size.count(clone))
by sergio.rabellino＠unito.it 06 May '24

06 May '24

Dear Ceph users, I'm pretty new on this list, but I've been using Ceph with satisfaction since 2020. I faced some problems through these years consulting the list archive, but now we're down with a problem that seems without an answer. After a power failure, we have a bunch of OSDs that during rebalance/refilling goes down with this error: /build/ceph-OM2K9O/ceph-13.2.9/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fdcb2523700 time 2024-05-02 17:18:40.680350 /build/ceph-OM2K9O/ceph-13.2.9/src/osd/osd_types.cc: 5084: FAILED assert(clone_size.count(clone)) ceph version 13.2.9 (58a2a9b31fd08d8bb3089fce0e312331502ff945) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7fdcd38f63ee] 2: (()+0x287577) [0x7fdcd38f6577] 3: (SnapSet::get_clone_bytes(snapid_t) const+0x125) [0x555e697c2725] 4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2c8) [0x555e696d8208] 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x1169) [0x555e6973f749] 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x1018) [0x555e69743b98] 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x36a) [0x555e695b07da] 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x555e69813c99] 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x52d) [0x555e695b220d] 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x476) [0x7fdcd38fc516] 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7fdcd38fd6d0] 12: (()+0x76db) [0x7fdcd23e46db] 13: (clone()+0x3f) [0x7fdcd13ad61f] -6171> 2024-05-02 17:18:40.680 7fdcb2523700 -1 *** Caught signal (Aborted) ** in thread 7fdcb2523700 thread_name:tp_osd_tp And we're unable to understand what's happening. Yes, actually we're in Luminous but we planned to upgrade to Pacific in June, but before upgrading I believe it's important to have a positive health check. The pools in error are EC pools. Some hints ?

2 1

CLT meeting notes May 6th 2024
by Adam King 06 May '24

06 May '24

- DigitalOcean credits - things to ask - what would promotional material require - how much are credits worth - Neha to ask - 19.1.0 centos9 container status - close to being ready - will be building centos 8 and 9 containers simultaneously - should test on orch and upgrade suites before publishing RC - should first RC be tested on LRC? - skip first RC on LRC - performance differences to 18.2.2 with cephfs being investigated - 18.2.3 - fix for https://tracker.ceph.com/issues/65733 almost ready - will upgrade LRC to version with fix - will happen before 19.1.0 most likely

1 0

Off-Site monitor node over VPN
by Stefan Pinter 06 May '24

06 May '24

Hi! i hope someone can help us out here :) We need to move from 3 datacenters to 2 datacenters (+ 1 small serverroom reachable via layer 3 VPN) NOW we have a ceph-mon in each datacenter, which is fine. But we have to move and will only have 2 datacenters in the future (that are connected, so devices that are separated geographically can still communicate over their VLANs that they are used to have) in order to still have 3 - to continue being quorate - we have the idea to only move 1 ceph-mon into a small separate datacenter. we only are able to connect it via VPN though and we'd need a separate IP network and the traffic would be over a Firewall and over VPN. having a separate IP network should work, as mentioned here: 2.1.1.2<https://2.1.1.2/> Monitoring nodes on different subnets https://documentation.suse.com/ses/7/html/ses-all/storage-bp-hwreq.html what could possibly go wrong here? can the added latency be a problem? which devices would need to be allowed to connect over VPN? All OSDs < - vpn - > ceph-mon thank you for any input! Kind regards Stefan ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 0

radosgw sync non-existent bucket ceph reef 18.2.2
by Christopher Durham 06 May '24

06 May '24

Hi, I have a reef cluster 18.2.2 on Rocky 8.9. This cluster has been upgraded from pacific->quincy->reef over the past few years. It is a multi site with one other cluster that works fine with s3/radosgw on both sides, with proper bidirectional data replication. On one of the master cluster's radosgw logs, I noticed a sync request regarding a deleted bucket. I am not sure when this error started, but I know that the bucket in question was deleted a long time before the upgrade to reef. Perhapsthis error existed prior to reef, I do not know. Here is the error in the radosgw log: :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe{s={b=BUCKET_NAME:CLUSTERID ..., z=...., az= ...},d={b=..,az=...}} returned ret=-2 My understanding: s=source, d=destination, each of which is a tuple with the appropriate info necessary.... This happens for BUCKET_NAME every few minutes. Said bucket does not exist on either side of the multisite, but did in the past. Any way I can force radosgw to stop trying to replicate? Thanks -Chris

2 1

Unable to add new OSDs
by ceph＠mikesoffice.com 06 May '24

06 May '24

I'm trying to add a new storage host into a Ceph cluster (quincy 17.2.6). The machine has boot drives, one free SSD and 10 HDDs. The plan is to have each HDD be an OSD with a DB on a equal size lvm of the SDD. This machine is newer but otherwise similar to other machines already in the cluster that are setup and running the same way. But I've been unable to add OSDs and unable to figure out why, or fix it. I have some experience, but I'm not an expert and could be missing something obvious. If anyone has any suggestions, I would appreciate it. I've tried to add OSDs a couple different ways. Via the dashboard, this has worked fine for previous machines. And it appears to succeed and gives no errors that I can find looking in /var/log/ceph and dashboard logs. The OSDs are never created. In fact, the drives still show up as available in Physical Disks and I can do the same creation procedure repeatedly. I've tried creating it in cephadm shell with the following, which has also worked in the past: ceph orch daemon add osd stor04.fqdn:data_devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi,/dev/sdj,/dev/sdk,db_devices=/dev/sda,osds_per_device=1 The command just hangs. Again I wasn't able to find any obvious errors. Although this one did seem to cause some slow op errors from the monitors that required restarting a monitor. And it could cause errors with the dashboard locking up and having to restart the manager as well. And I've tried setting 'ceph orch apply osd --all-available-devices --unmanaged=false' to let Ceph automatically add the drives. In the past, this would cause Ceph to automatically add the drives as OSDs but without having associated DBs on the SSD. The SSD would just be another OSD. This time it appears to have no affect and similar to the above, I wasn't able to find any obvious error feedback. -Mike

4 3

Have a problem with haproxy/keepalived/ganesha/docker
by Ruslan Nurabayev 06 May '24

06 May '24

Hello! I've installed my 5-node CEPH cluster next install NFS server by command: ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip 192.168.171.48/26 --ingress-mode haproxy-protocol. I don't understand fully how this must be works but when i stop NFS daemon even on one of this nodes I've see that writing on NFS shares is disappear (testing via vdbench). As i understand it is wrong and IO from stopped daemon must switching to another NFS daemon without any impact on IO. Can someone help me with troubleshoot this issue? Or explain how done full-fledged Active-Active HA NFS Cluster for production use. Thanks! Руслан Нурабаев Старший инженер Сектор ИТ платформы Отдел развития опорной сети Департамент развития сети +77012119272 Ruslan.Nurabayev(a)kcell.kz -----Original Message----- From: ceph-users-request(a)ceph.io <ceph-users-request(a)ceph.io> Sent: Thursday, April 11, 2024 15:07 To: Ruslan Nurabayev <Ruslan.Nurabayev(a)kcell.kz> Subject: Welcome to the "ceph-users" mailing list [You don't often get email from ceph-users-request(a)ceph.io. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Welcome to the "ceph-users" mailing list! To post to this list, send your email to: ceph-users(a)ceph.io You can unsubscribe or make adjustments to your options via email by sending a message to: ceph-users-request(a)ceph.io with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI. ________________________________ **************************************************************************************** Осы хабарлама және онымен берілетін кез келген файлдар құпия болып табылады және олар мекенжайда көрсетілген жеке немесе заңды тұлғалардың пайдалануына ғана арналған. Егер сіз болжамды алушы болып табылмайтын болсаңыз, осы арқылы осындай ақпаратты кез келген таратуға, жіберуге, көшіруге немесе пайдалануға қатаң тыйым салынатыны және осы электрондық хабарлама дереу жойылуға тиіс екендігін хабарлаймыз. KCELL осы хабарламадағы кез келген ақпараттың дәлдігіне немесе толықтығына қатысты ешқандай кепілдік бермейді және сол арқылы онда қамтылған ақпарат үшін немесе оны беру, қабылдау, сақтау немесе қандай да бір түрде пайдалану үшін кез келген жауапкершілікті болдырмайды. Осы хабарламада айтылған пікірлер тек жіберушіге ғана тиесілі және KCELL пікірін де білдіруі міндетті емес. Бұл электрондық хабарлама барлық танымал компьютерлік вирустарға тексерілді. **************************************************************************************** Данное сообщение и любые передаваемые с ним файлы являются конфиденциальными и предназначены исключительно для использования физическими или юридическими лицами, которым они адресованы. Если вы не являетесь предполагаемым получателем, настоящим уведомляем о том, что любое распространение, пересылка, копирование или использование такой информации строго запрещено, и данное электронное сообщение должно быть немедленно удалено. KCELL не дает никаких гарантий относительно точности или полноты любой информации, содержащейся в данном сообщении, и тем самым исключает любую ответственность за информацию, содержащуюся в нем, или за ее передачу, прием, хранение или использование каким-либо образом. Мнения, выраженные в данном сообщении, принадлежат только отправителю и не обязательно отражают мнение KCELL. Данное электронное сообщение было проверено на наличие всех известных компьютерных вирусов. **************************************************************************************** This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted. KCELL makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, reception, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of KCELL. This e-mail has been scanned for all known computer viruses. ****************************************************************************************

6 8

Remove failed OSD
by Zakhar Kirpichenko 04 May '24

04 May '24

Hi! An OSD failed in our 16.2.15 cluster. I prepared it for removal and ran `ceph orch daemon rm osd.19 --force`. Somehow that didn't work as expected, so now we still have osd.19 in the crush map: -10 122.66965 host ceph02 19 1.00000 osd.19 down 0 1.00000 But OSD has been cleaned up on the host, although incompletely, as both block and block.db LVs still exist. If I try to remove the OSD again, I get an error: # ceph orch daemon rm osd.19 --force Error EINVAL: Unable to find daemon(s) ['osd.19'] How can I clean up this OSD and get rid of it completely, including the crush map? I would appreciate any suggestions or pointers. Best regards, Zakhar

2 2

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Reset health.
by Albert Shih 03 May '24

03 May '24

Hi, Very basic question : 2 days ago I reboot all the cluster. Everything work fine. But I'm guessing during the shutdown 4 osd was mark as crash [WRN] RECENT_CRASH: 4 daemons have recently crashed osd.381 crashed on host cthulhu5 at 2024-03-20T18:33:12.017102Z osd.379 crashed on host cthulhu4 at 2024-03-20T18:47:13.838839Z osd.376 crashed on host cthulhu3 at 2024-03-20T18:50:00.877536Z osd.373 crashed on host cthulhu1 at 2024-03-20T18:56:46.887394Z is they are any way to «clean» that ? Because otherwise my icinga complain.... I don't like to add a downtime in icinga. Thanks. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 22 mars 2024 22:24:35 CET

3 2

cephadm custom crush location hooks
by Eugen Block 03 May '24

03 May '24

Hi, we've been using custom crush location hooks for some OSDs [1] for years. Since we moved to cephadm, we always have to manually edit the unit.run file of those OSDs because the path to the script is not mapped into the containers. I don't want to define custom location hooks for all OSDs globally in the OSD spec, even if those are limited to two hosts only in our case. But I'm not aware of a method to target only specific OSDs to have some files mapped into the container [2]. Is my assumption correct that we'll have to live with the manual intervention until we reorganize our osd tree? Or did I miss something? Thanks! Eugen [1] https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-location… [2] https://docs.ceph.com/en/latest/cephadm/services/#mounting-files-with-extra…

2 8

'ceph fs status' no longer works?
by Erich Weiler 03 May '24

03 May '24

Hi All, For a while now I've been using 'ceph fs status' to show current MDS active servers, filesystem status, etc. I recently took down my MDS servers and added RAM to them (one by one, so the filesystem stayed online). After doing that with my four MDS servers (I had two active and two standby), all looks OK, 'ceph -s' reports HEALTH_OK. But when I do 'ceph fs status' now, I get this: # ceph fs status Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status assert metadata AssertionError This is on ceph 18.2.1 reef. This is very odd - can anyone think of a reason why 'ceph fs status' would stop working after taking each of the servers down for maintenance? The filesystem is online and working just fine however. This ceph instance is deployed via the cephadm method on RHEL 9.3, so the everything is containerized in podman. Thanks again, erich

4 6

NVME node disks maxed out during rebalance after adding to existing cluster
by Szabo, Istvan (Agoda) 02 May '24

02 May '24

Hi, I have slow heartbeat in front and back with the extra node added to the cluster and this occasionally causing slow ops and failed osd reports. I'm extending our cluster with +3 relatively differently configured servers compared to the original 12. Our cluster (latest octopus) is an objectstore cluster with 12x identical node (8x15.3 TB SSD inside with 4osd on each, 512GB mem, 96 vcore cpu ...) and it hosts a 4:2 EC data pool. The +3 nodes - currently done only 1 and now the 2nd is in progress but have the issue during rebalance - have 8x 15.3TB NVME drives with 4x osd on each. The NVME drive specification:https://i.ibb.co/BVmLKnf/currentnvme.png [https://i.ibb.co/BVmLKnf/currentnvme.png] The old server SSD spec: https://i.ibb.co/dkD3VKx/oldssd.png [https://i.ibb.co/dkD3VKx/oldssd.png] Iostat on new nvme: https://i.ibb.co/PF0hrVW/iostat.png [https://i.ibb.co/PF0hrVW/iostat.png] Rebalance is ongoing with the slowest option like max_backfill/op priority/max recovery = 1 But it generates a very huge iowait, seems like the disk not fast enough (but why in the previous node didn't have this issue)? Here is the metrics about the disk which is running the backfill/rebalance now (FYI we have 3.2B objects in the cluster): https://i.ibb.co/LNXCRbj/disks.png [https://i.ibb.co/LNXCRbj/disks.png] Wonder what I'm missing or how this can happen? Here you can see gigantic latencies, failed osds and slow ops: https://i.ibb.co/Jn0sj9g/laten.png [https://i.ibb.co/Jn0sj9g/laten.png] Thank you for your help ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Reconstructing an OSD server when the boot OS is corrupted
by Peter van Heusden 02 May '24

02 May '24

Dear Ceph Community We have 5 OSD servers running Ceph v15.2.17. The host operating system is Ubuntu 20.04. One of the servers has suffered corruption to its boot operating system. Using a system rescue disk it is possible to mount the root filesystem but it is not possible to boot the operating system at the moment. The OSDs are configured with (spinning disk) data drives, WALs and DBs on partitions of SSDs, but from my examination of the filesystem the configuration in /var/lib/ceph appears to be corrupted. So my question is: what is the best option for repair going forward? Is it possible to do a clean install of the operating system and scan the existing drives in order to reconstruct the OSD configuration? Thank you, Peter P.S. the cause of the original corruption is likely due to an unplanned power outage, an event that hopefully will not recur.

7 9

service:mgr [ERROR] "Failed to apply:
by Roberto Maggi ＠ Debian 02 May '24

02 May '24

Hi you all, it is a couple of days I'm facing this problem. Although I already destroyed the cluster a couple of times I continuously get these error I instruct ceph to place 3 daemons ceph orch apply mgr 3 --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" but root@cephstage01:/# ceph -s cluster: id: 0827effc-302a-45a6-bb97-758d2a9e6835 health: HEALTH_ERR Failed to apply 6 service(s): alertmanager,grafana,mgr,mon,node-exporter,prometheus failed to probe daemons or devices 2 mgr modules have failed 1 mgr modules have recently crashed OSD count 0 < osd_pool_default_size 3 services: mon: 2 daemons, quorum cephstage01,cephstagedatanode01 (age 21m) mgr: cephstage01.ukwnlo(active, since 5h), standbys: cephstagedatanode01.vqbwbz osd: 0 root@cephstage01:/# ceph orch ls --service_name mgr --format yaml service_type: mgr service_name: mgr placement: hosts: - cephstage01:10.20.20.81 - cephstage02:10.20.20.82 - cephstage03:10.20.20.83 status: created: '2024-05-02T13:52:41.346547Z' last_refresh: '2024-05-02T14:06:31.823024Z' running: 2 size: 1 events: - 2024-05-02T13:51:49.993145Z service:mgr [INFO] "service was created" - '2024-05-02T13:53:45.566950Z service:mgr [ERROR] "Failed to apply: Cannot place <ServiceSpec for service_name=mgr> on cephstage02, cephstage03: Unknown hosts"' and when I check those hosts root@cephstage01:/# ceph cephadm check-host cephstage03 cephstage03 (None) ok docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chrony.service is enabled and running Hostname "cephstage03" matches what is expected. Host looks OK they look fine. every time I recreate the cluster it gives me the same errors: does anybody have any idea? thanks in advance

2 1

Ceph Day NYC 2024 Slides
by Matt Vandermeulen 02 May '24

02 May '24

Hi folks! Thanks for a great Ceph Day event in NYC! I wanted to make sure I posted my slides before I forget (and encourage others to do the same). Feel free to reach out in the Ceph Slack https://ceph.io/en/community/connect/ How we Operate Ceph at Scale (DigitalOcean): - https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… - https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… Discards in Ceph (DigitalOcean): - https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… - https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… Matt

2 2

After dockerized ceph cluster to Pacific, the fsid changed in the output of 'ceph -s'
by wjsherry075＠outlook.com 02 May '24

02 May '24

Hello, I had a problem after I finished the 'cephadm adopt' from services to docker containers for mon and mgr. The fsid of `ceph -s` is not the same as the /etc/ceph/ceph.conf. The ceph.conf is correct, but `ceph -s` is incorrect. I followed the https://docs.ceph.com/en/quincy/cephadm/adoption/ ``` 2024-04-25T19:49:17.460652+0000 mgr.cloud-lab-test-mon01 (mgr.65113) 109 : cephadm [ERR] cephadm exited with an error code: 1, stderr:ERROR: fsid does not match ceph.conf Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:ERROR: fsid does not match ceph.conf #ceph health detail shows the warning: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices ``` Does anyone else have any ideas? Thanks, Sherry

2 1

Ceph client cluster compatibility
by Nima AbolhassanBeigi 02 May '24

02 May '24

Hi, We are trying to upgrade our OS version from ubuntu 18.04 to ubuntu 22.04. Our ceph cluster version is 16.2.13 (pacific). The problem is that the ubuntu packages for the ceph pacific release will not be supported for ubuntu 22.04. We were wondering if the ceph client (version 18.2, reef) on ubuntu 22.04 can work with lower version clusters. Thanks in advance Regards Nima

2 1

RGW multisite slowness issue due to the "304 Not Modified" responses on primary zone
by praveenkumargpk17＠gmail.com 01 May '24

01 May '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our Grafana instance and out of 1000 requests / second, 200 requests are getting "200 OK" and 800 requests are getting "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks, Praveen

4 4