July 2023 - ceph-users - lists.ceph.io

User + Dev Monthly Meeting happening tomorrow

by Laura Flores

Hi everyone, The User + Dev Monthly Meeting is happening tomorrow, July 20th at 2:00 PM UTC at this link: https://meet.jit.si/ceph-user-dev-monthly Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

10 months

1
1
0 0

Ceph Leadership Team Meeting, 2023-07-19 Minutes

by Patrick Donnelly

Hi folks, Today we discussed: - Reef is almost ready! The remaining issues are tracked in [1]. In particular, an epel9 package is holding back the release. - Vincent Hsu, Storage Group CTO of IBM, presented a proposal outline for a Ceph Foundation Client Council. This council would be composed of 10-25 invited significant operators or users of Ceph. The function of the council is to provide essential feedback on use-cases, pain-points, and successes arising during their use of Ceph. This feedback will be used to steer development and initiatives. More information on this will be forthcoming once the proposal is finalized. The monthly user <-> dev meeting will be reevaluated in light of this, possibly continuing on as usual. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

10 months

1
1
0 0

RBD image QoS rbd_qos_write_bps_limit and rbd_qos_bps_limit and mkfs performance

by Engelmann Florian

Hi, I noticed an incredible high performance drop with mkfs.ext4 (as well as mkfs.xfs) when setting (almost) "any" value for rbd_qos_write_bps_limit (or rbd_qos_bps_limit). Baseline: 4TB rbd volume rbd_qos_write_bps_limit = 0 mkfs.ext4: real 0m6.688s user 0m0.000s sys 0m0.006s 50GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 53687091200 mkfs.ext4: real 1m22.217s user 0m0.009s sys 0m0.000s 5GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 5368709120 mkfs.ext4: real 13m39.770s user 0m0.008s sys 0m0.034s 500MB/s: 4TB rbd volume rbd_qos_write_bps_limit = 524288000 mkfs.ext4: test still runing... I can provide the result if needed. The tests are running on a client vm (Ubuntu 22.04) using Qemu/libvirt. Using the same values with Qemu/libvirt QoS does not affect mkfs performance. https://libvirt.org/formatdomain.html#block-i-o-tuning Ceph Version: 16.2.11 Qemu: 6.2.0 Libvirt: 8.0.0 Kernel (hypervisor host): 5.19.0-35-generic librbd1 (hypervisor host): 17.2.5 Could anyone pls confirm and explain what's going on? All the best, Florian

10 months

2
3
0 0

librbd hangs during large backfill

by fb2cd0fc-933c-4cfe-b534-93d67045a088＠simplelogin.com

Starting on Friday, as part of adding a new pod of 12 servers, we initiated a reweight on roughly 384 drives; from 0.1 to 0.25. Something about the resulting large backfill is causing librbd to hang, requiring server restarts. The volumes are showing buffer i/o errors when this happens.We are currently using hybrid OSDs with both SSD and traditional spinning disks. The current status of the cluster is: ceph --version ceph version 14.2.22 Cluster Kernel 5.4.49-200 { "mon": { "ceph version 14.2.22 nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.22 nautilus (stable)": 3 }, "osd": { "ceph version 14.2.21 nautilus (stable)": 368, "ceph version 14.2.22 (stable)": 2055 }, "mds": {}, "rgw": { "ceph version 14.2.22 (stable)": 7 }, "overall": { "ceph version 14.2.21 (stable)": 368, "ceph version 14.2.22 (stable)": 2068 } } HEALTH_WARN, noscrub,nodeep-scrub flag(s) set. pgs: 6815703/11016906121 objects degraded (0.062%) 2814059622/11016906121 objects misplaced (25.543%). The client servers are on 3.10.0-1062.1.2.el7.x86_6 We have found a couple of issues that look relevant: https://tracker.ceph.com/issues/19385 https://tracker.ceph.com/issues/18807 Has anyone experienced anything like this before? Does anyone have any recommendations as to settings that can help alleviate this while the backfill completes? An example of the buffer ii/o errors: Jul 17 06:36:08 host8098 kernel: buffer_io_error: 22 callbacks suppressed Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 0, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-4, logical block 3, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-5, logical block 511984, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-6, logical block 3487657728, async page read Jul 17 06:36:08 host8098 kernel: Buffer I/O error on dev dm-6, logical block 3487657729, async page read

10 months

4
4
0 0

1 PG stucked in "active+undersized+degraded for long time

by siddhit.renake＠nxtgen.com

Hello All, Ceph version: 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable) Issue: 1 PG stucked in "active+undersized+degraded for long time Degraded data redundancy: 44800/8717052637 objects degraded (0.001%), 1 pg degraded, 1 pg undersized #ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 15.28f0 active+undersized+degraded [2147483647,343,355,415,426,640,302,392,78,202,607] 343 [2147483647,343,355,415,426,640,302,392,78,202,607] 343 PG Query: #ceph pg 15.28f0 query { "state": "active+undersized+degraded", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 303362, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting_recovery_backfill": [ "78(8)", "202(9)", "302(6)", "343(1)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "info": { "pgid": "15.28f0s1", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598853, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "303161'598853", "reported_seq": "3594551", "reported_epoch": "303362", "state": "active+undersized+degraded", "last_fresh": "2023-06-20 19:03:59.135295", "last_change": "2023-06-20 15:11:12.569114", "last_active": "2023-06-20 19:03:59.135295", "last_peered": "2023-06-20 19:03:59.135295", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2023-06-20 15:11:12.569114", "last_became_peered": "2023-06-20 15:11:12.569114", "last_unstale": "2023-06-20 19:03:59.135295", "last_undegraded": "2023-06-20 15:11:10.430426", "last_fullsized": "2023-06-20 15:11:10.430154", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3028, "ondisk_log_size": 3028, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54989065178, "num_objects": 44800, "num_object_clones": 0, "num_object_copies": 492800, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44800, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44800, "num_whiteouts": 0, "num_read": 201078, "num_read_kb": 30408632, "num_write": 219335, "num_write_kb": 59459782, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44800 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "78(8)", "pgid": "15.28f0s8", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "202(9)", "pgid": "15.28f0s9", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "302(6)", "pgid": "15.28f0s6", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "355(2)", "pgid": "15.28f0s2", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "392(7)", "pgid": "15.28f0s7", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "415(3)", "pgid": "15.28f0s3", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "426(4)", "pgid": "15.28f0s4", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "607(10)", "pgid": "15.28f0s10", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "640(5)", "pgid": "15.28f0s5", "last_update": "303161'598853", "last_complete": "303161'598853", "log_tail": "261289'595825", "last_user_version": 598852, "last_backfill": "MAX", "last_backfill_bitwise": 1, "purged_snaps": [], "history": { "epoch_created": 19841, "epoch_pool_created": 16141, "last_epoch_started": 303017, "last_interval_started": 303016, "last_epoch_clean": 250583, "last_interval_clean": 250582, "last_epoch_split": 19841, "last_epoch_marked_full": 0, "same_up_since": 303016, "same_interval_since": 303016, "same_primary_since": 256311, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439" }, "stats": { "version": "302924'598852", "reported_seq": "3594069", "reported_epoch": "302924", "state": "active+undersized+degraded", "last_fresh": "2023-06-19 20:33:19.103885", "last_change": "2021-04-28 20:51:18.679339", "last_active": "2023-06-19 20:33:19.103885", "last_peered": "2023-06-19 20:33:19.103885", "last_clean": "2021-04-11 15:21:44.271834", "last_became_active": "2021-04-28 20:51:18.679339", "last_became_peered": "2021-04-28 20:51:18.679339", "last_unstale": "2023-06-19 20:33:19.103885", "last_undegraded": "2021-04-28 20:51:18.617123", "last_fullsized": "2021-04-28 20:51:18.616747", "mapping_epoch": 303016, "log_start": "261289'595825", "ondisk_log_start": "261289'595825", "created": 19841, "last_epoch_clean": 250583, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "255277'537760", "last_scrub_stamp": "2021-04-11 03:18:39.164439", "last_deep_scrub": "255277'537756", "last_deep_scrub_stamp": "2021-04-10 01:42:16.182528", "last_clean_scrub_stamp": "2021-04-11 03:18:39.164439", "log_size": 3027, "ondisk_log_size": 3027, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 54984870874, "num_objects": 44799, "num_object_clones": 0, "num_object_copies": 492789, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 44799, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 44799, "num_whiteouts": 0, "num_read": 201036, "num_read_kb": 30408590, "num_write": 219334, "num_write_kb": 59455686, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 121970, "num_bytes_recovered": 73212503327, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "acting": [ 2147483647, 343, 355, 415, 426, 640, 302, 392, 78, 202, 607 ], "avail_no_missing": [ "343(1)", "78(8)", "202(9)", "302(6)", "355(2)", "392(7)", "415(3)", "426(4)", "607(10)", "640(5)" ], "object_location_counts": [ { "shards": "78(8),202(9),302(6),343(1),355(2),392(7),415(3),426(4),607(10),640(5)", "objects": 44799 } ], "blocked_by": [], "up_primary": 343, "acting_primary": 343, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 303017, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2023-06-20 15:11:10.430274", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } }, "scrub": { "scrubber.epoch_start": "0", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2023-06-20 15:11:09.294876" } ], "agent_state": {} } Require assistance to to make this PG active+clean

10 months

3
4
0 0

Another Pacific point release?

by Ponnuvel Palaniyappan

Hi, The typical EOL date (2023-06-01) has already passed for Pacific. Just wondering if there's going to be another Pacific point release (16.2.14) in the pipeline. -- Regards, Ponnuvel P

10 months

4
3
0 0

replacing all disks in a stretch mode ceph cluster

by Zoran Bošnjak

Hello ceph users, my ceph configuration is - ceph version 17.2.5 on ubuntu 20.04 - stretch mode - 2 rooms with OSDs and monitors + additional room for the tiebreaker monitor - 4 OSD servers in each room - 6 OSDs per OSD server - ceph installation/administration is manual (without ansible, orch... or any other tool like this) Ceph health is currently OK. Raw usage is around 60%, Pools usage is below 75% I need to replace all OSD disks in the cluster with larger capacity disks (500G to 1000G). So the eventual configuration will contain the same number of OSDs and servers. I understand I can replace OSDs one by one, following the documented procedure (removing old and adding new OSD to the configuration) and waiting for health OK. But in this case, ceph will probably copy data around like crazy after each step. So, my question is: What is the recommended procedure in this case of replacing ALL disks and keeping the ceph operational during the upgrade? In particular: Should I use any of "nobackfill, norebalance, norecover..." flags during the process? If yes, which? Should I do one OSD at the time, server at the time or even room at the time? Thanks for the suggestions. regards, Zoran

10 months

3
2
0 0

Workload that delete 100 M object daily via lifecycle

by Ha Nguyen Van

Hi Experts, We plan to setup a Ceph Object to support a S3 workload, that will need to delete 100M file daily via lifecycle. Appreciate your suggestion and setting to handle this kind of scenario Best Regards, Ha

10 months

4
5
0 0

Re: librbd hangs during large backfill

by Jack Hayhurst

We did have a peering storm, we're past that portion of the backfill and still experiencing new instances of rbd volumes hanging. It is for sure not just the peering storm. We've got 22.184% objects misplaced yet, with a bunch of pgs left to backfill (like 75k). Our rbd poll is using about 1.7PiB of storage, so we're looking at like 370TiB yet to backfill, rough estimate. This specific pool is using replicated encoding, with size=3. RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 21 PiB 11 PiB 10 PiB 10 PiB 48.73 TOTAL 21 PiB 11 PiB 10 PiB 10 PiB 48.73 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL pool1 4 32768 574 TiB 147.16M 1.7 PiB 68.87 260 TiB We did see a lot of rbd volumes that hung, often giving the buffer i/o errors previously sent - whether that was the peering storm or backfills is uncertain. As suggested, we've already been detaching/reattaching the rbd volumes, pushing the primary active osd for pgs to another, and sometimes rebooting the kernel on the vm to clear the io queue. A combination of those brings the rbd volume block device back for a while. We're no longer in a peering storm and we're seeing the rbd volumes going into an unresponsive state again - including osds where they were unresponsive, we did things and got them responsive, and then they went unresponsive again. All pgs are in an active state, some active+remapped+backfilling, some active+undersized+remapped+backfilling, etc. We also run the object gateway off the same cluster with the same backfill, the object gateway is not experiencing issues. Also the osds patricipating in the backfill are not saturated with i/o, or seeing abnormal load for our usual backfill operations. But with the continuing backfill, we're seeing rbd volumes on active pgs going back into a blocked state. We can do about the same with detaching the volume / bouncing the pg to a new primary acting osd, but we'd rather have these stop going unresponsive in the first place. Any suggestions towards that direction are greatly appreciated.

10 months

1
0
0 0

CEPHADM_FAILED_SET_OPTION

by Arnoud de Jonge

Hi, After having set up RadosGW with keystone authentication the cluster shows this warning: # ceph health detail HEALTH_WARN Failed to set 1 option(s) [WRN] CEPHADM_FAILED_SET_OPTION: Failed to set 1 option(s) Failed to set rgw.fra option rgw_keystone_implicit_tenants: config set failed: error parsing value: 'True' is not one of the permitted values: false, true, swift, s3, both, 0, 1, none retval: -22 I might have made a typo but that has been corrected. # ceph config dump |grep rgw_keystone_implicit_tenants client.rgw.fra.controller1.lushdc advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller1.pznmuf advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller1.tdrqot advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.ndxaet advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.rodqxh advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller2.wyhjuk advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.boasgr advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.lkczbl advanced rgw_keystone_implicit_tenants true * client.rgw.fra.controller3.qxctee advanced rgw_keystone_implicit_tenants true * # ceph --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Any idea on how to fix this issue? Thanks, Arnoud.

10 months

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users July 2023