- ceph-users - lists.ceph.io

bluefs_buffered_io
by Marcel Kuiper 04 Nov '20

04 Nov '20

Hi list, I see a few changes in the (minor) version changelogs in the default for bluefs_buffered_io setting. Sometimes it is set to true in our version (14.2.11) it is set to false Can someone shed a light on this setting? I fail to find any documentation on it. ceph config help is not entirely clear to me as well - What does it do exactly when true - If false does that mean that the linux buffer cache is always skipped? And caching happens in the osd proces only? - if enabled should we lower the osd_memory_target to leave more space for the linux buffer cache? What would be the percentage of memory that we then assign to osd_memory_targets Marcel

1 0

Ceph 14.2 - some PGs stuck peering.
by m.sliwinski＠lh.pl 04 Nov '20

04 Nov '20

Hi We have a weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting any errors. Story: We have 3 different pools, hdd-backed, ssd-backed and nvme-backed. Pool ssh worked fine for few months. Today one of the hosts assigned to nvme pool restarted triggering recovery in that pool. It wnet fast and cluster went to OK state. During these events or shortly after them ssd pool became unresponsive. It was impossible to either read or write from/to it. We decided to slowly restart fist OSDs assigned to it, thenm as it didn't help - all the mons, wihout breaking quorum of course. At this moment both nvme and hdd polls are working fine, ssd one is stuck in recovery. All OSDs in that ssd pool use large amount of CPU and are exchanging approx 1Mpps per OSD server between each other. PGs seem to be slowly migrating from peering to activating but it's going very slowly - approx 10PGs during last hour. We were using 14.2.2 OSDs when issues happened, upgrade to 14.2.13 didn't help. We increased heartbeat grace, but it didn't change anything. It doesn't seem that there's a network problem as OSDs don't report problems with connecting to MONs or each other. Other OSDs - nvme, connected to that same set of switches work without issues. Can you help? Point me to what should i check or do? I looked on-line and on the group for causes of peering issues and checked most of them, nothing helped. I can't use 'ceph pg 28.1cc query' as it hangs, even for PGs that are marked as active+clean in the results of 'ceph pg dump' I checked status of the one of stuck PGs via ceph-objectstore-tool --data-path [...] --op info --pgid 28.29d for all three copies and got: { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69447, "same_interval_since": 69447, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378496", "reported_epoch": "69447", "state": "peering", "last_fresh": "2020-11-03 20:55:39.247348", "last_change": "2020-11-03 20:55:39.247348", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 20:55:39.247348", "last_undegraded": "2020-11-03 20:55:39.247348", "last_fullsized": "2020-11-03 20:55:39.247348", "mapping_epoch": 69447, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271, 284 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69422, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69630, "same_interval_since": 69630, "same_primary_since": 69628, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378445", "reported_epoch": "69627", "state": "peering", "last_fresh": "2020-11-03 21:15:08.819278", "last_change": "2020-11-03 21:14:18.360957", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 21:15:08.819278", "last_undegraded": "2020-11-03 21:15:08.819278", "last_fullsized": "2020-11-03 21:15:08.819278", "mapping_epoch": 69630, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284 ], "acting": [ 261, 284 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69392, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69411, "same_interval_since": 69411, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68070'205093", "reported_seq": "378344", "reported_epoch": "68160", "state": "active+clean", "last_fresh": "2020-11-03 14:45:02.988293", "last_change": "2020-11-03 03:32:46.896044", "last_active": "2020-11-03 14:45:02.988293", "last_peered": "2020-11-03 14:45:02.988293", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-09-01 13:52:40.091759", "last_unstale": "2020-11-03 14:45:02.988293", "last_undegraded": "2020-11-03 14:45:02.988293", "last_fullsized": "2020-11-03 14:45:02.988293", "mapping_epoch": 69411, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3093, "ondisk_log_size": 3093, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 67746, "hit_set_history": { "current_last_update": "0'0", "history": [] } } Current status of the cluster: Reduced data availability: 1021 pgs inactive, 999 pgs peering Degraded data redundancy: 18357/94939584 objects degraded (0.019%), 3 pgs degraded, 5 pgs undersized services: mon: 3 daemons, quorum monb01,monb02,monb03 mgr: monb03(active), standbys: monb01, monb02 osd: 285 osds: 284 up, 284 in data: pools: 9 pools, 9546 pgs objects: 31.65 M objects, 120 TiB usage: 363 TiB used, 127 TiB / 490 TiB avail pgs: 10.696% pgs not active 18357/94939584 objects degraded (0.019%) 8520 active+clean 999 peering 18 activating 3 active+clean+scrubbing+deep 2 activating+undersized+degraded 2 activating+undersized 1 active+clean+scrubbing 1 active+undersized+degraded io: client: 367 MiB/s rd, 195 MiB/s wr, 24.51 kop/s rd, 5.95 kop/s wr cache: 24 MiB/s flush, 90 MiB/s evict, 23 op/s promote

2 2

Re: Seriously degraded performance after update to Octopus
by Martin Rasmus Lundquist Hansen 04 Nov '20

04 Nov '20

Thank you for the suggestion. It does indeed seem to explain why the OSD nodes are no longer using the Buffers for caching. Unfortunately, changing the value bluefs_buffered_io does not seem to make any difference in performance. I will keep looking for clues.

1 0

How to reset Log Levels
by Ml Ml 04 Nov '20

04 Nov '20

Hello, i played around with some log level i can´t remember and my logs are now getting bigger than my DVD-Movie collection. E.g.: journalctl -b -u ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)mon.ceph03.service > out.file is 1,1GB big. I did already try: ceph tell mon.ceph03 config set debug_mon 0/10 ceph tell mon.ceph03 config set debug_osd 0/10 ceph tell mon.ceph03 config set debug_mgr 0/10 ceph tell mon.ceph03 config set "mon_health_to_clog" false ceph tell mon.ceph03 config set "mon_health_log_update_period" 30 ceph tell mon.ceph03 config set "debug_mgr" "0/0" which made it better, but i really cant remember it all and would like to have the default values. Is there a way to reset those Log Values? Cheers, Michael

2 2

Ceph 14.2 - stuck peering.
by m.sliwinski＠lh.pl 04 Nov '20

04 Nov '20

Hi We have weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting any errors. Story: We have 3 different pools, hdd-backed, ssd-backed and nvme-backed. Pool ssh worked fine for few months. Today one of the hosts assigned to nvme pool restarted triggering recovery in that pool. It wnet fast and cluster went to OK state. During these events or shortly after them ssd pool became unresponsive. It was impossible to either read or write from/to it. We decided to slowly restart fist OSDs assigned to it, thenm as it didn't help - all the mons, wihout breaking quorum of course. At this moment both nvme and hdd polls are working fine, ssd one is stuck in recovery. All OSDs in that ssd pool use large amount of CPU and are exchanging approx 1Mpps per OSD server between each other. PGs seem to be slowly migrating from peering to activating but it's going very slowly - approx 10PGs during last hour. We were using 14.2.2 OSDs when issues happened, upgrade to 14.2.13 didn't help. We increased heartbeat grace, but it didn't change anything. It doesn't seem that there's a network problem as OSDs don't report problems with connecting to MONs or each other. Other OSDs - nvme, connected to that same set of switches work without issues. Can you help? Point me to what should i check or do? I looked on-line and on the group for causes of peering issues and checked most of them, nothing helped. I can't use 'ceph pg 28.1cc query' as it hangs, even for PGs that are marked as active+clean in the results of 'ceph pg dump' I checked status of the one of stuck PGs via ceph-objectstore-tool --data-path [...] --op info --pgid 28.29d for all three copies and got: { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69447, "same_interval_since": 69447, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378496", "reported_epoch": "69447", "state": "peering", "last_fresh": "2020-11-03 20:55:39.247348", "last_change": "2020-11-03 20:55:39.247348", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 20:55:39.247348", "last_undegraded": "2020-11-03 20:55:39.247348", "last_fullsized": "2020-11-03 20:55:39.247348", "mapping_epoch": 69447, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271, 284 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69422, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69630, "same_interval_since": 69630, "same_primary_since": 69628, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378445", "reported_epoch": "69627", "state": "peering", "last_fresh": "2020-11-03 21:15:08.819278", "last_change": "2020-11-03 21:14:18.360957", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 21:15:08.819278", "last_undegraded": "2020-11-03 21:15:08.819278", "last_fullsized": "2020-11-03 21:15:08.819278", "mapping_epoch": 69630, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284 ], "acting": [ 261, 284 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69392, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69411, "same_interval_since": 69411, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68070'205093", "reported_seq": "378344", "reported_epoch": "68160", "state": "active+clean", "last_fresh": "2020-11-03 14:45:02.988293", "last_change": "2020-11-03 03:32:46.896044", "last_active": "2020-11-03 14:45:02.988293", "last_peered": "2020-11-03 14:45:02.988293", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-09-01 13:52:40.091759", "last_unstale": "2020-11-03 14:45:02.988293", "last_undegraded": "2020-11-03 14:45:02.988293", "last_fullsized": "2020-11-03 14:45:02.988293", "mapping_epoch": 69411, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3093, "ondisk_log_size": 3093, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 67746, "hit_set_history": { "current_last_update": "0'0", "history": [] } } Current status of the cluster: Reduced data availability: 1021 pgs inactive, 999 pgs peering Degraded data redundancy: 18357/94939584 objects degraded (0.019%), 3 pgs degraded, 5 pgs undersized services: mon: 3 daemons, quorum monb01,monb02,monb03 mgr: monb03(active), standbys: monb01, monb02 osd: 285 osds: 284 up, 284 in data: pools: 9 pools, 9546 pgs objects: 31.65 M objects, 120 TiB usage: 363 TiB used, 127 TiB / 490 TiB avail pgs: 10.696% pgs not active 18357/94939584 objects degraded (0.019%) 8520 active+clean 999 peering 18 activating 3 active+clean+scrubbing+deep 2 activating+undersized+degraded 2 activating+undersized 1 active+clean+scrubbing 1 active+undersized+degraded io: client: 367 MiB/s rd, 195 MiB/s wr, 24.51 kop/s rd, 5.95 kop/s wr cache: 24 MiB/s flush, 90 MiB/s evict, 23 op/s promote

1 0

Inconsistent Space Usage reporting
by Vikas Rana 03 Nov '20

03 Nov '20

Hi Friends, We have some inconsistent storage space usage reporting. We used only 46TB with single copy but the space used on the pool is close to 128TB. Any idea where's the extra space is utilized and how to reclaim it? Ceph Version : 12.2.11 with XFS OSDs. We are planning to upgrade soon. # ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 363TiB 131TiB 231TiB 63.83 43.80M POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED fcp 15 N/A N/A 23.6TiB 42.69 31.7TiB 3053801 3.05M 6.10GiB 12.6GiB 47.3TiB nfs 16 N/A N/A 128TiB 66.91 63.4TiB 33916181 33.92M 3.93GiB 4.73GiB 128TiB # df -h Filesystem Size Used Avail Use% Mounted on /dev/nbd0 200T 46T 155T 23% /vol/dir_research #ceph osd pool get nfs all size: 1 min_size: 1 crash_replay_interval: 0 pg_num: 128 pgp_num: 128 crush_rule: replicated_ruleset hashpspool: true nodelete: false nopgchange: false nosizechange: false write_fadvise_dontneed: false noscrub: false nodeep-scrub: false use_gmt_hitset: 1 auid: 0 fast_read: 0 Appreciate your help. Thanks, -Vikas

2 3

Restart Error: osd.47 already exists in network host
by Ml Ml 03 Nov '20

03 Nov '20

Hello List, sometimes some OSD get taken our for some reason ( i am still looking for the reason, and i guess its due to some overload), however, when i try to restart them i get: Nov 02 08:05:26 ceph05 bash[9811]: Error: No such container: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df-osd.47 Nov 02 08:05:29 ceph05 bash[9811]: /usr/bin/docker: Error response from daemon: endpoint with name ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df-osd.47 already exists in network host. Nov 02 08:05:29 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Main process exited, code=exited, status=125/n/a Nov 02 08:05:34 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Failed with result 'exit-code'. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Service RestartSec=10s expired, scheduling restart. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Scheduled restart job, restart counter is at 5. Nov 02 08:05:44 ceph05 systemd[1]: Stopped Ceph osd.47 for 5436dd5d-83d4-4dc8-a93b-60ab5db145df. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Start request repeated too quickly. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Failed with result 'exit-code'. Nov 02 08:05:44 ceph05 systemd[1]: Failed to start Ceph osd.47 for 5436dd5d-83d4-4dc8-a93b-60ab5db145df. I need to reboot the full host to get the OSD back in again. As far i can see this is some docker problem? root@ceph05:~# docker ps | grep osd.47 => no hit root@ceph05:~# docker network prune => does not solve the problem Any hint on that? Thanks, Michael

2 3

Re: How to recover from active+clean+inconsistent+failed_repair? [SOLVED]
by Sagara Wijetunga 03 Nov '20

03 Nov '20

Hi Frank Found the issue and fixed. It was a one copy of 0 byte object. Removed it. Deep scrub the PG fixed the issue. # find /var/lib/ceph/osd/ -type f -name "1000023675e*" /var/lib/ceph/osd/ceph-2/current/3.b_head/DIR_B/DIR_A/DIR_E/1000023675e.00000000__head_AE97EEAB__3 # ls -l /var/lib/ceph/osd/ceph-2/current/3.b_head/DIR_B/DIR_A/DIR_E/1000023675e.00000000__head_AE97EEAB__3-rw-r--r-- 1 ceph ceph 0 Oct 31 19:18 /var/lib/ceph/osd/ceph-2/current/3.b_head/DIR_B/DIR_A/DIR_E/1000023675e.00000000__head_AE97EEAB__3 Once again, many thanks for your help. Best regards Sagara

2 1

Updating client caps online
by Dan van der Ster 03 Nov '20

03 Nov '20

Hi all, We still have legacy caps on our nautilus rbd cluster. I just wanted to check if this is totally safe (and to post here ftr because I don't think this has ever been documented) Here are the current caps: [client.images] key = xxx caps mgr = "allow r" caps mon = "allow r, allow command \"osd blacklist\"" caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=images" [client.volumes] key = xxx caps mgr = "allow r" caps mon = "allow r, allow command \"osd blacklist\"" caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images, allow rwx pool=cinder-critical" Now that we upgraded to nautilus we would do: # ceph auth caps client.images mon 'profile rbd' osd 'profile rbd pool=images' mgr 'profile rbd pool=images' # ceph auth caps client.volumes mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd-read-only pool=images, profile rbd pool=cinder-critical' mgr 'profile rbd pool=volumes, profile rbd pool=cinder-critical' Does that look correct? Does this apply without impacting any client IOs ? Thanks! Dan

2 2

Does it make sense to have separate HDD based DB/WAL partition
by huxiaoyu＠horebdata.cn 03 Nov '20

03 Nov '20

Dear Ceph folks, I am faced with a special use Ceph scenario, where no SSD is available. Three node Ceph cluster, each with 14 HDDs, no SSD. In this case, should i create a separate DB/WAL partition for each OSD on a HDD? and if yes, how big the partition? Which is the best in terms of performance, with a separate DB/WAL partition, or without? Thanks a lot, samuel huxiaoyu(a)horebdata.cn

2 1