- ceph-users - lists.ceph.io

Put Bucket Logging | Method Not Allowed
by Wissem MIMOUNA - Ceph Users 05 Jun '25

05 Jun '25

Hi Ceph Users, I'm trying to test ceph radosgw bucket logging feature , but I keep getting "MethodNotAllowed" Error, though I reinstalled the radosgw from scratch (I recreated a new realm-zonegroup and zone). Our ceph cluster is ceph squid v 19.2.2 on debian 12. I followed the procedure in the following url to create a bucket policy (https://docs.ceph.com/en/latest/radosgw/bucket_logging/) I tried both , with normal users and with account users. Each time the the user/account who created the both bucket ( source bucket and logging destination bucket) who try to enable logging but failed. The command which keep failing is (I tried with boto3 and with aws cli but the same error)==> 3cmd accesslog s3://myBucket --access-logging-target-prefix=logsBucket/prefix Any advices are welcome! Thanks Wissem M

3 2

MDS Repeatedly Crashing/Restarting - Unable to get CephFS Active
by Kasper Rasmussen 05 Jun '25

05 Jun '25

Ceph Version: 18.2.7 I've just migrated to cephadm, and upgrade from pacific to reef 18.2.7 last week. All successful except some minor issues with BlueFS Spillover Today the MDS of a specific fs refuse to start, and the ceph orch ps shows the daemons with status "error". I have three other cephfs that still works(though I haven't tested if they can fail over.) I've restartet the MDSs - No luck (the selected MDS just start/crash in a loop until it gives up) I've deployed 2 new MDSs - No luck same issue In all scenarios I see in ceph fs status, that a MDS is chosen. FS status goes to "replay" or "replay(laggy)" On the host with the MDS I see the MDS container just crashes after way less than 5 mins.. And status reported by ceph orch ps is error. (btw - mds_beacon_grace has been set to 360) I've managed to get a good 500 lines of log out with info like this: << ----------------- LOG EXAMPLE START ----------------- >> -7> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_tickets -6> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2025-05-19T16:04:32.845551+0000) -5> 2025-05-19T16:05:02.860+0000 7f673e3c1640 10 monclient: get_auth_request con 0x5616e9616c00 auth_method 0 -4> 2025-05-19T16:05:02.916+0000 7f673dbc0640 10 monclient: get_auth_request con 0x5616e7422800 auth_method 0 -3> 2025-05-19T16:05:02.968+0000 7f673d3bf640 10 monclient: get_auth_request con 0x5616f5eac800 auth_method 0 -2> 2025-05-19T16:05:02.972+0000 7f6736bb2640 2 mds.0.cache Memory usage: total 574800, rss 343772, heap 207124, baseline 182548, 0 / 7535 inodes have caps, 0 caps, 0 caps per inode -1> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f67333ab640 time 2025-05-19T16:05:03.680495+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f67406e6d2c] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 3: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 4: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 5: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 6: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 7: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 8: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 9: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 10: clone() 0> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 *** Caught signal (Aborted) ** in thread 7f67333ab640 thread_name:mds-log-replay ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f674004bbf0] 2: /lib64/libc.so.6(+0x8bf5c) [0x7f6740098f5c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f67406e6d86] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 7: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 8: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 10: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 11: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 12: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 13: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 14: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. << ----------------- LOG EXAMPLE END ----------------- >> But to be honest, out of all those lines, I don't know what to provide (all +500 might be a bit to much) I really need this FS back online, so help will be very much appreciated

4 14

Help in upgrading CEPH
by Sergio Rabellino 05 Jun '25

05 Jun '25

Dear list, we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow: > # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too). It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous: > "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag. All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us. Any hints ? thanks in advance. -- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>

5 16

Write throughput increasing on CephFS metadata pool
by Adrien Georget 05 Jun '25

05 Jun '25

Hi, We have a CephFS Quincy cluster (17.2.7) used by Openshift for PVC provisioning (and lot of snapshots) with ceph-csi driver. On the metadata pool, we observed a continuous increase of the write throughput activity from 10MB/s just after a restart of the MDS to +200MB/s after 2 weeks. We first thought it was related to the snapshots deletion activity (lot of creations and deletions every day) because we saw the same kind of increase for the number of Strays. We tried to evaluate strays using the recursive scrub [1], Strays decrease from 450K to 350K but no impact on the metadata write throughput. Only a MDS restart makes the write throughput back to normal. I saw this very familiar issue https://tracker.ceph.com/issues/53542 and I'm wondering if this has been fixed in Quincy? I tried to increase "mds_log_events_per_segment" and "mds_log_max_segments" but nothing helped. As I answered in the tracker, I also observed large 4M objects in objecter_requests. Anything we can do to prevent this and avoid restarting the MDS every week? If we don't do that, PVC operations stay blocked on Openshift side. Cheers, Adrien [1] https://docs.ceph.com/en/quincy/cephfs/scrub/#evaluate-strays-using-recursi… The MDS perf dump if it can helps : { "AsyncMessenger::Worker-0": { "msgr_recv_messages": 452889868, "msgr_send_messages": 210932005, "msgr_recv_bytes": 381554341884, "msgr_send_bytes": 31897539519042, "msgr_created_connections": 5392, "msgr_active_connections": 213, "msgr_running_total_time": 121964.941943159, "msgr_running_send_time": 85193.342708858, "msgr_running_recv_time": 40274.249544605, "msgr_running_fast_dispatch_time": 15899.702855460, "msgr_send_messages_queue_lat": { "avgcount": 210932003, "sum": 140532.121408536, "avgtime": 0.000666243 }, "msgr_handle_ack_lat": { "avgcount": 110313532, "sum": 120.513184629, "avgtime": 0.000001092 } }, "AsyncMessenger::Worker-1": { "msgr_recv_messages": 393334601, "msgr_send_messages": 170841678, "msgr_recv_bytes": 307649876710, "msgr_send_bytes": 23017109865662, "msgr_created_connections": 12712, "msgr_active_connections": 218, "msgr_running_total_time": 93136.265793790, "msgr_running_send_time": 63604.143090523, "msgr_running_recv_time": 30881.136884349, "msgr_running_fast_dispatch_time": 12763.126679902, "msgr_send_messages_queue_lat": { "avgcount": 170841678, "sum": 153574.265735747, "avgtime": 0.000898927 }, "msgr_handle_ack_lat": { "avgcount": 81594240, "sum": 65.480936528, "avgtime": 0.000000802 } }, "AsyncMessenger::Worker-2": { "msgr_recv_messages": 481956104, "msgr_send_messages": 301348856, "msgr_recv_bytes": 349838769013, "msgr_send_bytes": 26783354654792, "msgr_created_connections": 34394, "msgr_active_connections": 215, "msgr_running_total_time": 108807.807447203, "msgr_running_send_time": 74577.666991790, "msgr_running_recv_time": 41392.667149426, "msgr_running_fast_dispatch_time": 13971.142479134, "msgr_send_messages_queue_lat": { "avgcount": 301348847, "sum": 138655.862052108, "avgtime": 0.000460117 }, "msgr_handle_ack_lat": { "avgcount": 156375610, "sum": 144.168097813, "avgtime": 0.000000921 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "queue_len": 0, "complete_latency": { "avgcount": 121564865, "sum": 160601.142778677, "avgtime": 0.001321114 } }, "finisher-PurgeQueue": { "queue_len": 0, "complete_latency": { "avgcount": 27338289, "sum": 8785.146872773, "avgtime": 0.000321349 } }, "mds": { "request": 210298249, "reply": 210297956, "reply_latency": { "avgcount": 210297956, "sum": 1148337.019419176, "avgtime": 0.005460523 }, "slow_reply": 15, "forward": 0, "dir_fetch": 49843551, "dir_commit": 65281430, "dir_split": 6, "dir_merge": 17, "inodes": 5113817, "inodes_top": 2080514, "inodes_bottom": 2263929, "inodes_pin_tail": 769374, "inodes_pinned": 2141653, "inodes_expired": 538454667, "inodes_with_caps": 413837, "caps": 457168, "subtrees": 2, "traverse": 262113637, "traverse_hit": 245551660, "traverse_forward": 0, "traverse_discover": 0, "traverse_dir_fetch": 1642726, "traverse_remote_ino": 0, "traverse_lock": 140893, "load_cent": 392174, "q": 0, "exported": 0, "exported_inodes": 0, "imported": 0, "imported_inodes": 0, "openino_dir_fetch": 407633, "openino_backtrace_fetch": 7011755, "openino_peer_discover": 0, "root_rfiles": 11618539, "root_rbytes": 11106041675168, "root_rsnaps": 7576, "scrub_backtrace_fetch": 7009788, "scrub_set_tag": 0, "scrub_backtrace_repaired": 0, "scrub_inotable_repaired": 0, "scrub_dir_inodes": 1526050, "scrub_dir_base_inodes": 3, "scrub_dirfrag_rstats": 1526047, "scrub_file_inodes": 5483733, "handle_inode_file_caps": 0, "ceph_cap_op_revoke": 5076186, "ceph_cap_op_grant": 60192925, "ceph_cap_op_trunc": 658164, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_flush_ack": 6, "handle_client_caps": 70349707, "handle_client_caps_dirty": 21547109, "handle_client_cap_release": 11468771, "process_request_cap_release": 83631146 }, "mds_cache": { "num_strays": 347989, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 19259955, "strays_enqueued": 20275628, "strays_reintegrated": 469, "strays_migrated": 0, "num_recovering_processing": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "recovery_started": 484, "recovery_completed": 484, "ireq_enqueue_scrub": 2, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 23, "ireq_fragstats": 0, "ireq_inodestats": 0 }, "mds_log": { "evadd": 148604638, "evex": 148687537, "evtrm": 148686957, "ev": 4640, "evexg": 0, "evexd": 580, "segadd": 5143165, "segex": 5143142, "segtrm": 5143128, "seg": 166, "segexg": 0, "segexd": 14, "expos": 2655552712972056, "wrpos": 2655556750517207, "rdpos": 2574717722176006, "jlat": { "avgcount": 23948614, "sum": 1026045.477642289, "avgtime": 0.042843626 }, "replayed": 86959 }, "mds_mem": { "ino": 5091233, "ino+": 537662754, "ino-": 532571521, "dir": 385511, "dir+": 48473070, "dir-": 48087559, "dn": 5114534, "dn+": 564239550, "dn-": 559125016, "cap": 457198, "cap+": 296065469, "cap-": 295608271, "rss": 21947020, "heap": 223516 }, "mds_server": { "dispatch_client_request": 265192774, "dispatch_server_request": 0, "handle_client_request": 210298249, "handle_client_session": 44456292, "handle_peer_request": 0, "req_create_latency": { "avgcount": 19045215, "sum": 243874.341572325, "avgtime": 0.012805019 }, "req_getattr_latency": { "avgcount": 13244393, "sum": 122372.637049784, "avgtime": 0.009239580 }, "req_getfilelock_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_link_latency": { "avgcount": 108, "sum": 0.089209875, "avgtime": 0.000826017 }, "req_lookup_latency": { "avgcount": 42591076, "sum": 61153.611919024, "avgtime": 0.001435831 }, "req_lookuphash_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupino_latency": { "avgcount": 6119, "sum": 12.051986148, "avgtime": 0.001969600 }, "req_lookupname_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupparent_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupsnap_latency": { "avgcount": 11789, "sum": 1.916955778, "avgtime": 0.000162605 }, "req_lssnap_latency": { "avgcount": 174999, "sum": 59.628852495, "avgtime": 0.000340738 }, "req_mkdir_latency": { "avgcount": 386018, "sum": 5323.027974258, "avgtime": 0.013789584 }, "req_mknod_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_mksnap_latency": { "avgcount": 11784, "sum": 4342.774640033, "avgtime": 0.368531452 }, "req_open_latency": { "avgcount": 1719118, "sum": 2957.961208770, "avgtime": 0.001720627 }, "req_readdir_latency": { "avgcount": 49303877, "sum": 153658.488737674, "avgtime": 0.003116559 }, "req_rename_latency": { "avgcount": 1021332, "sum": 4173.200914205, "avgtime": 0.004086037 }, "req_renamesnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_rmdir_latency": { "avgcount": 355348, "sum": 6609.629921457, "avgtime": 0.018600442 }, "req_rmsnap_latency": { "avgcount": 11476, "sum": 4647.171801363, "avgtime": 0.404947002 }, "req_rmxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setattr_latency": { "avgcount": 951092, "sum": 60918.695526648, "avgtime": 0.064051317 }, "req_setdirlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setfilelock_latency": { "avgcount": 42236246, "sum": 87407.285106607, "avgtime": 0.002069485 }, "req_setlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setxattr_latency": { "avgcount": 20450454, "sum": 351779.625311317, "avgtime": 0.017201555 }, "req_symlink_latency": { "avgcount": 2, "sum": 0.002286490, "avgtime": 0.001143245 }, "req_unlink_latency": { "avgcount": 18777510, "sum": 39044.878444925, "avgtime": 0.002079342 }, "cap_revoke_eviction": 0, "cap_acquisition_throttle": 0, "req_getvxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "mds_sessions": { "session_count": 545, "session_add": 26598, "session_remove": 26053, "sessions_open": 545, "sessions_stale": 0, "total_load": 11076, "average_load": 20, "avg_session_uptime": 37409721, "mdthresh_evicted": 0 }, "mempool": { "bloom_filter_bytes": 10309112, "bloom_filter_items": 10309112, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_items": 0, "bluefs_bytes": 0, "bluefs_items": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "buffer_anon_bytes": 284731060, "buffer_anon_items": 5959770, "buffer_meta_bytes": 616, "buffer_meta_items": 7, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 81480, "osdmap_items": 2396, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 16277298996, "mds_co_items": 260156681, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "op_active": 11, "op_laggy": 0, "op_send": 253406063, "op_send_bytes": 81014361089067, "op_resend": 0, "op_reply": 253406052, "oplen_avg": { "avgcount": 253406063, "sum": 451959672 }, "op": 253406063, "op_r": 58269533, "op_w": 195136530, "op_rmw": 0, "op_pg": 0, "osdop_stat": 64974069, "osdop_create": 11505623, "osdop_read": 1275145, "osdop_write": 34882891, "osdop_writefull": 1169760, "osdop_writesame": 0, "osdop_append": 0, "osdop_zero": 2, "osdop_truncate": 4, "osdop_delete": 79836916, "osdop_mapext": 0, "osdop_sparse_read": 0, "osdop_clonerange": 0, "osdop_getxattr": 56700621, "osdop_setxattr": 17474636, "osdop_cmpxattr": 0, "osdop_rmxattr": 0, "osdop_resetxattrs": 0, "osdop_call": 0, "osdop_watch": 0, "osdop_notify": 0, "osdop_src_cmpxattr": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_other": 683839, "linger_active": 0, "linger_send": 0, "linger_resend": 0, "linger_ping": 0, "poolop_active": 0, "poolop_send": 0, "poolop_resend": 0, "poolstat_active": 0, "poolstat_send": 0, "poolstat_resend": 0, "statfs_active": 0, "statfs_send": 0, "statfs_resend": 0, "command_active": 0, "command_send": 0, "command_resend": 0, "map_epoch": 685274, "map_full": 0, "map_inc": 19880, "osd_sessions": 96, "osd_session_open": 96, "osd_session_close": 0, "osd_laggy": 0, "omap_wr": 78737471, "omap_rd": 99825478, "omap_del": 4893217 }, "oft": { "omap_total_objs": 4, "omap_total_kv_pairs": 105334, "omap_total_updates": 80286291, "omap_total_removes": 72648222 }, "purge_queue": { "pq_executing_ops": 0, "pq_executing_ops_high_water": 1559, "pq_executing": 0, "pq_executing_high_water": 64, "pq_executed": 20275628, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "val": 0, "max": 104857600, "get_started": 0, "get": 1328180571, "get_sum": 935681276002, "get_or_fail_fail": 0, "get_or_fail_success": 1328180571, "take": 0, "take_sum": 0, "put": 1328180571, "put_sum": 935681276002, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_bytes": { "val": 21734389, "max": 104857600, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 253406063, "take_sum": 81151894362586, "put": 253406052, "put_sum": 81151872628197, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_ops": { "val": 11, "max": 1024, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 253406063, "take_sum": 253406063, "put": 253406052, "put_sum": 253406052, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle": { "val": 0, "max": 3758096384, "get_started": 0, "get": 20275628, "get_sum": 2230259884, "get_or_fail_fail": 0, "get_or_fail_success": 20275628, "take": 0, "take_sum": 0, "put": 1242717, "put_sum": 2230259884, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle-0x56401344c0a0": { "val": 0, "max": 3758096384, "get_started": 0, "get": 148604638, "get_sum": 80839025768070, "get_or_fail_fail": 0, "get_or_fail_success": 148604638, "take": 0, "take_sum": 0, "put": 23948617, "put_sum": 80839025768070, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } } }

2 2

radosgw force usage of sse-kms?
by Boris 05 Jun '25

05 Jun '25

Hi, I am in a project where the requirement is: - All data MUST be encrypted at rest - At least one key per customer - The backend only supports the vault KV as secrect engine, so SSE-s3 is not an option The idea is, that we create a key for the customer and tell them in the panel that they need to use it and how it will work. But how to I prevent the user from uploading unecrypted objects? Do I check for a header in the proxy and return a <h1>uhuhuh, you didn't say the magic word!</h1> when there specific header is missing? And if this is the way, is there a shema I need to stick to? - Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 3

Ceph S3 Performance: HDDs for data + SSDs for metadata – Best practice?
by Ml Ml 04 Jun '25

04 Jun '25

Hi everyone, I'm currently planning a Ceph-based S3 storage backend, with a strong focus on balancing performance and cost (EUR per TB). Fully SSD-based setups seem too expensive for my use case, while HDD-only configurations might be too slow. My current idea is to use a hybrid approach: - HDDs for bluestore_block (bulk data) - Enterprise SSDs for bluestore_block.db and bluestore_block.wal (metadata and journal) The main client of this S3 backend will be Veeam, which, as far as I can tell, uses larger block sizes. Since this results in fewer random writes and mostly sequential I/O, I assume this kind of workload can work well with HDDs — as long as metadata is fast (hence SSDs for DB/WAL). Am I on the right track with this assumption? - Do you have any benchmark results (real-world or synthetic) comparing HDD-only vs Hybrid setups for object workloads like Veeam? That would help a lot to quantify the performance difference. - What SSD-to-HDD ratio would you recommend? (I’ve seen 1 SSD per 3–5 HDDs) Any advice or confirmation from those running similar setups would be much appreciated! So far i found: https://www.hyperscalers.com/image/catalog/00-Products/Storage%20Servers/Wh… => which mentions Ceph Verstion 0.94.5 and the performance boot does not seem too big here And here: https://www.ambedded.com.tw/en/use-case/use-case-08.html they dont mention such a Hybrid Setup Thanks a lot in advance! Best regards, Michael

4 3

radosgw-admin bucket radoslist returns 1.8 billion entries for a bucket with 17m objects
by Boris 04 Jun '25

04 Jun '25

Hi, I am in the process of checking orphan objects and the radoslist list grep over 300GB while the rados ls list was only 50GB. After some investigation I identified a saw that one bucket fell out with a lot of objects in the radoslist part. I did some sorting on the output file and just used the marker part to check if something is wrong: These are the last two lines after sorting: 16145185 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644274.205 1882589930 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.1805769661.3772 The top bucket has roughtly the same amount ob objects in the index, but the other bucket only hast 17million object in the index. I stopped the search for orphan objects and pulled a fresh `radosgw-admin bucket radoslist --bucket BUCKET > file`. Then I sorted that list and ran a uniq -c against it and sorted that list again. The unique list ist now 34million objects long and there are 32026 entries that appear more than once and there are entries that 68849 times. 68849 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.1805769661.3772__shadow_server/download/XXX/YYYY.2~k9UuEdtUkAHkikXz1ONc8UzgwKryYco.6_1 How can I clean that up? Or is this just a bug in the radosgw-admin bucket radoslist? The bucket has 401 shards and got "versioned: true" and "versioning_enabled: true" -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

Spurious CephFS backtrace objects in non-default pool
by Hector Martin 04 Jun '25

04 Jun '25

Hi, I just noticed an interesting behavior that I didn't find documented anywhere. Normally, CephFS stores backtraces in the both default data pool and the actual data pool a file's layout points to (if different): $ for pool in cephfs2_data cephfs2_data_hr3 cephfs2_data_hec5.4; do echo "# Pool: $pool" rados -p $pool listxattr 100032d5676.00000000 done # Pool: cephfs2_data parent # Pool: cephfs2_data_hr3 error getting xattr set cephfs2_data_hr3/100032d5676.00000000: (2) No such file or directory # Pool: cephfs2_data_hec5.4 layout parent However, I found that some files have backtrace objects in another pool: $ for pool in cephfs2_data cephfs2_data_hr3 cephfs2_data_hec5.4; do echo "# Pool: $pool" rados -p $pool listxattr 100032c464a.00000000 done # Pool: cephfs2_data parent # Pool: cephfs2_data_hr3 parent # Pool: cephfs2_data_hec5.4 layout parent Apparently, this happens when you create a file in a directory that is assigned to a non-default pool, and then change the pool before writing any data into the object. In other words, any pool that a file is *ever* assigned to after being created empty, ends up with a backtrace object. This is visible in the backtrace object itself: rados -p cephfs2_data getxattr 100032c464a.00000000 parent \ | head -c -1 \ | ceph-dencoder type inode_backtrace_t import - decode dump_json { "ino": 1099564861002, "ancestors": [ <snip> ], "pool": 14, "old_pools": [ 11, 12 ] } Here pool 11 is the default pool, pool 14 is the desired pool, and pool 12 is the pool of the directory layout for the directory the file was created in (before being reassigned prior to writing data). I found this rather surprising, as I'm migrating a large amount of data to a new pool by setting file layouts before writing out data to the new temporary file [1] and my temporary directory happened to be assigned to a non-default pool, so now I have a bunch of garbage objects in that pool (which happens to be an HDD pool, so this hurts performance). I would have expected the directory's assigned layout pool to be irrelevant if I override it after opening a new file for writing, but apparently this is not the case. I guess this is so existing clients can find the real pool if they have the old pool cached or something? I'm not entirely sure why this behavior exists. It would be nice if there were some way to clean this up without having to rewrite all the files again (it's terabytes of data). I don't mind if I have to take the FS down for a bit to run a cleanup tool, but doing manual surgery on the backtrace objects and inode omap values seems pretty difficult and error-prone :/ [1] https://gist.github.com/marcan/26cc3ac7241f866dca38916215dd10ff - Hector

2 5

PG inconsistent one extra object?
by Work Ceph 03 Jun '25

03 Jun '25

Hello guys, We are facing an unexpected error with a single PG after a node addition. We are seeing the following error message: ``` scrub : stat mismatch, got 299586/299585 objects, 0/0 clones, 299586/299585 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 366757864470/366755158108 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. log_channel(cluster) log [ERR] scrub 1 error ``` It looks as if there is an extra object in the PG. Have you guys ever seen such a situation? The command "rados list-inconsistent-snapset" does not return anything. The command "rados list-inconsistent-pg" only returns the affected PG. The command "rados list-inconsistent-obj" does not return anything. We are suspecting that this might be caused by the number of PGs in the pool, which is too small. However, before trying to increase it, we are trying to understand what is going on. Is it safe to increase the number of PGs with a PG inconsistent with this situation with an extra object? We are running Ceph on Ubuntu 22. We are using the standard Ceph binaries from the distro.

2 5

Application for Official Ceph Mirror – Telepoint, Bulgaria
by mirror＠telepoint.bg 03 Jun '25

03 Jun '25

Hello, I am contacting you to apply to add our mirror as an official mirror of the Ceph project. Our mirror is hosted at Telepoint DataCenter in Sofia, Bulgaria. We have carefully reviewed and complied with all the official Ceph mirror requirements, including logging policies, synchronization frequency, directory structure, and protocol support. Here are the technical and administrative details: Domains: - Primary: mirror.telepoint.bg - Alternative: mirror.bg IP Addresses: - IPv4: 78.128.80.69, 78.128.80.70 - IPv6: 2a01:8740:ffff:fffc::31, 2a01:8740:ffff:fffc::32 Available Protocols: - HTTP: http://mirror.telepoint.bg/ceph/ - HTTPS: https://mirror.telepoint.bg/ceph/ - RSYNC: rsync://mirror.telepoint.bg/ceph/ Other Details: - Sync schedule: Every 3 hours - Bandwidth: 40 Gbps - Sponsor: Telepoint.bg - Location: Sofia, Bulgaria - Contact email: mirror(a)telepoint.bg We would like to request the CNAME bg.ceph.com to point to mirror.telepoint.bg. Thank you for your time and consideration. Please let us know if you need any additional information. -- Best regards, Valentin Nikolov Telepoint Mirror Service 122, Ovche Pole str. 1303 Sofia, Bulgaria Phone: +359 2 490 32 10 Mobile: +359 893 590 195 Mobile 2: +359 888 536 968 Fax: +359 2 490 3212 E-mail: noc(a)telepoint.bg web: https://www.telepoint.bg telepoint.bg

1 0