- ceph-users - lists.ceph.io

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

How to make config changes stick for MDS?
by Erich Weiler 17 Apr '24

17 Apr '24

Hi All, I'm having a crazy time getting config items to stick on my MDS daemons. I'm running Reef 18.2.1 on RHEL 9 and the daemons are running in podman, I used cephadm to deploy the daemons. I can adjust the config items in runtime, like so: ceph tell mds.slugfs.pr-md-01.xdtppo config set mds_bal_interval -1 But for the life of me I cannot get that to stick when I restart the MDS daemon. I've tried adding this to /etc/ceph/ceph.conf in the host server: [mds] mds_bal_interval = -1 But that doesn't get picked up on daemon restart. I also added the same config segment to /etc/ceph/ceph.conf *inside* the container, no dice, still doesn't stick. I even tried adding it to /var/lib/ceph/<uuid>/config/ceph.conf and it *still* doesn't stick across daemon restarts. Does anyone know how I can get MDS config items to stick across daemon reboots when the daemon is running in podman under RHEL? Thanks much! -erich

2 1

Ceph Leadership Team Meeting, 2024-04-08
by Laura Flores 16 Apr '24

16 Apr '24

Hi all, Today we discussed: 2024/04/08 - [Zac] CQ#4 is going out this week - https://pad.ceph.com/p/ceph_quarterly_2024_04 - Last chance to review! - [Zac] IcePic Initiative - context-sensitive help - do we regard the docs as a part of the online help? - https://pad.ceph.com/p/2024_04_08_cephadm_context_sensitive_help - docs.ceph.com should be main source of truth; can link to this or reference it generally as "see docs.ceph.com" - Squid RC status - Blockers tracked in: https://pad.ceph.com/p/squid-upgrade-failures - rgw: topic changes merged to main, but introduced some test failures. account changes blocked on topics - Non-blocker for RC0 - centos 9 containerization (status unknown?) - Non-blocker for RC0 - Follow up with Dan / Guillaume - RADOS has one outstanding blocker awaiting QA - Failing to register new account at Ceph tracker - error 404. - Likely related to Redmine upgrade over the weekend - Pacific eol: - Action item: in https://docs.ceph.com/en/latest/releases/, move to "archived" - 18.2.3 - one or two PRs from cephfs left - Milestone: https://github.com/ceph/ceph/milestone/19 Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 4

Ceph Users Feedback Survey
by Neha Ojha 16 Apr '24

16 Apr '24

Hi everyone, On behalf of the Ceph Foundation Board, I would like to announce the creation of, and cordially invite you to, the first of a recurring series of meetings focused solely on gathering feedback from the users of Ceph. The overarching goal of these meetings is to elicit feedback from the users, companies, and organizations who use Ceph in their production environments. You can find more details about the motivation behind this effort in our user survey [1] that we highly encourage all of you to take. This is an extension of the Ceph User Dev Meeting with concerted focus on Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by Matt Leonard, Bloomberg), to start off with. We would like to kick off this series of meetings on March 21, 2024. The survey will be open until March 18, 2024. Looking forward to hearing from you! Thanks, Neha [1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLM…

1 2

Announcing go-ceph v0.27.0
by John Mulligan 16 Apr '24

16 Apr '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.27.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, rgw, and other subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Setting up Hashicorp Vault for Encryption with Ceph
by Michael Worsham 16 Apr '24

16 Apr '24

Is there a how-to document available on how to setup Hashicorp's Vault for Ceph, preferably in a HA state? Due to some encryption needs, we need to set up LUKS, OSD encryption AND Ceph bucket encryption as well. Yes, we know there will be a performance hit, but the encrypt-everything is a hard requirement for our business needs since we have government and healthcare-related contracts. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

2 1

Setting S3 bucket policies with multi-tenants
by Thomas Bennett 15 Apr '24

15 Apr '24

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any documentation that discusses this scenario. Does anyone know how to specify the Principle and Resource section of a policy.json file? Or any other configuration that I might be missing? I've tried some variations on Principal and Resource including and excluding tenant information, but not no luck yet. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]}, "Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"], "Resource": [ "arn:aws:s3:::*Tenant1/backups*" ] }] } I'm using s3cmd for testing, so: s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/ Returns: s3://backups/: Policy updated And then testing: s3cmd --config s3cfg.readwrite ls s3://backups/ ERROR: Access to bucket 'backups' was denied ERROR: S3 error: 403 (AccessDenied) Thanks, Tom

4 4

MDS Behind on Trimming...
by Erich Weiler 14 Apr '24

14 Apr '24

Hi All, I've been battling this for a while and I'm not sure where to go from here. I have a Ceph health warning as such: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 MDSs report slow requests 1 MDSs behind on trimming services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-01.jemmdf(active, since 3w), standbys: pr-md-02.emffhz mds: 1/1 daemons up, 2 standby osd: 46 osds: 46 up (since 9h), 46 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 260.72M objects, 466 TiB usage: 704 TiB used, 424 TiB / 1.1 PiB avail pgs: 1306 active+clean 4 active+clean+scrubbing+deep 3 active+clean+scrubbing io: client: 123 MiB/s rd, 75 MiB/s wr, 109 op/s rd, 1.40k op/s wr And the specifics are: # ceph health detail HEALTH_WARN 1 MDSs report slow requests; 1 MDSs behind on trimming [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.slugfs.pr-md-01.xdtppo(mds.0): 99 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.slugfs.pr-md-01.xdtppo(mds.0): Behind on trimming (13884/250) max_segments: 250, num_segments: 13884 That "num_segments" number slowly keeps increasing. I suspect I just need to tell the MDS servers to trim faster but after hours of googling around I just can't figure out the best way to do it. The best I could come up with was to decrease "mds_cache_trim_decay_rate" from 1.0 to .8 (to start), based on this page: https://www.suse.com/support/kb/doc/?id=000019740 But it doesn't seem to help, maybe I should decrease it further? I am guessing this must be a common issue...? I am running Reef on the MDS servers, but most clients are on Quincy. Thanks for any advice! cheers, erich

5 24

PG inconsistent
by Albert Shih 12 Apr '24

12 Apr '24

Hi everyone. I got a warning with root@cthulhu1:/etc/ceph# ceph -s cluster: id: 9c5bb196-c212-11ee-84f3-c3f2beae892d health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent So I find the pg with the issue, and launch a pg repair (still waiting) But I try to find «why» so I check all the OSD related on this pg and didn't find anything, no error from osd daemon, no errors from smartctl, no error from the kernel message. So I just like to know if that's «normal» or should I scratch deeper. JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 11:51:37 CEST

4 5

Issue about "ERROR: S3 Temporary Error: Request failed for: /. Please try again later"
by elite_stu＠163.com 11 Apr '24

11 Apr '24

I have mapped port 32505 to 23860, however when connect via s3cmd it fails with "ERROR: S3 Temporary Error: Request failed for: /. Please try again later." . has anyone ecounted same issue? [root@vm-04 ~]# s3cmd ls WARNING: Retrying failed request: / ('') WARNING: Waiting 3 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 6 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 9 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 12 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 15 sec... ERROR: S3 Temporary Error: Request failed for: /. Please try again later. [root@vm-04 ~]# [root@vm-04 ~]# cat .s3cfg [default] access_key = E9BJAC6QKLTOKVJR4TZC secret_key = KBvm0YqeAFtVM0I1gUhX5wrnwpHZZ5SLhiUa56ss host_base = xxxxxx:23860 host_bucket = xxxxxx:23860/bucket-king use_https = False signature_v2 = False [root@vm-04 ~]# bash-4.4$ ceph -s cluster: id: dd6a082e-868d-44e0-b93b-0f4956e5dd1a health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 4m) mgr: b(active, since 4m), standbys: a mds: 1/1 daemons up, 1 standby osd: 3 osds: 3 up (since 4m), 3 in (since 26m) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 233 pgs objects: 548 objects, 808 MiB usage: 2.4 GiB used, 148 GiB / 150 GiB avail pgs: 233 active+clean bash-4.4$ [root@vm-01 ~]# kubectl get svc -nrook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rook-ceph-mgr ClusterIP 10.105.165.62 <none> 9283/TCP 26m rook-ceph-mgr-dashboard ClusterIP 10.102.18.178 <none> 8443/TCP 26m rook-ceph-mgr-dashboard-external-https NodePort 10.100.66.254 <none> 8443:31091/TCP 24m rook-ceph-rgw-my-store ClusterIP 10.104.146.147 <none> 80/TCP 16m rook-ceph-rgw-my-store-external NodePort 10.106.39.241 <none> 80:32505/TCP 14m [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# kubectl get pods -nrook-ceph | grep rgw rook-ceph-rgw-my-store-a-67d579d99b-5xd64 2/2 Running 0 5m22s [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# kubectl -nrook-ceph logs rook-ceph-rgw-my-store-a-67d579d99b-5xd64 | tail -20 Defaulted container "rgw" out of: rgw, log-collector, chown-container-data-dir (init) debug 2024-04-11T13:42:33.635+0000 7fee46b43700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.630 +0000] "DELETE /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 204 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.005000210s debug 2024-04-11T13:42:33.635+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.647+0000 7fee4333c700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.012000503s ====== debug 2024-04-11T13:42:33.647+0000 7fee4333c700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.635 +0000] "PUT /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 HTTP/1.1" 200 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.012000503s debug 2024-04-11T13:42:33.648+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.666+0000 7fee3b32c700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.018000754s ====== debug 2024-04-11T13:42:33.666+0000 7fee3b32c700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.648 +0000] "PUT /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 200 21 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.018000754s debug 2024-04-11T13:42:33.667+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.669+0000 7fee4e352700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.002000083s ====== debug 2024-04-11T13:42:33.669+0000 7fee4e352700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.667 +0000] "GET /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 200 21 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.002000083s debug 2024-04-11T13:42:33.710+0000 7fee43b3d700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.719+0000 7fee2e312700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=204 latency=0.009000377s ====== debug 2024-04-11T13:42:33.719+0000 7fee2e312700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.710 +0000] "DELETE /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 204 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.009000377s debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 ====== starting new request req=0x7fee078c0730 ===== debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 ====== req done req=0x7fee078c0730 op status=0 http_status=200 latency=0.000000000s ====== debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 beast: 0x7fee078c0730: 192.168.100.8 - - [11/Apr/2024:13:42:37.863 +0000] "GET /swift/healthcheck HTTP/1.1" 200 0 - "kube-probe/1.28" - latency=0.000000000s debug 2024-04-11T13:42:41.078+0000 7fef3a54b700 0 rgw UsageLogger: WARNING: RGWRados::log_usage(): user name empty (bucket=), skipping debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 ====== starting new request req=0x7fee07a43730 ===== debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 ====== req done req=0x7fee07a43730 op status=0 http_status=200 latency=0.000000000s ====== debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 beast: 0x7fee07a43730: 192.168.100.8 - - [11/Apr/2024:13:42:47.862 +0000] "GET /swift/healthcheck HTTP/1.1" 200 0 - "kube-probe/1.28" - latency=0.000000000s [root@vm-01 ~]# [root@vm-01 ~]#

3 4