March 2021 - ceph-users - lists.ceph.io

Has anyone contact Data for Samsung Datacenter SSD Support ?
by Christoph Adomeit 16 Mar '21

16 Mar '21

Hi, I hope someone here can help me out with some contact data, email-adress or phone Number for Samsung Datacenter SSD Support ? If I contact Standard Samsung Datacenter Support they tell me they are not there to support PM1735 Drives. We are planning a new Ceph-Cluster and we are thinking of Samsung PM1735 NVME u.2 ssds Unfortunately the PM1735 is not available with u2 interface but the PM1733 is. Some manager from Samsung once told me that PM1733 and PM1735 is exactly the same Hardware, it is only provisioned differently. But he did not know whom to ask. Any idea of whom I could contact at Samsung or of how to provision the PM1733 (7.6TB) to a PM1735 (6.4TB) . I want the provisioning for better DWPD (3DWPD instead of 1DWPD).

3 2

RGW dashboard
by thomas.charles＠virttom.io 16 Mar '21

16 Mar '21

HI, this is regarding my fresh installation of CEPH on a Proxmox cluster. I need to use the radosGW and I'm quite stucked in dashboard configuration. ceph -v ceph version 15.2.9 (7b3df4a1b15c5a048c237733c797a2667f08196e) octopus (stable) ceph -w cluster: id: 86c332e8-9e83-4a11-bd10-7fc34d4fbd0a health: HEALTH_OK services: mon: 3 daemons, quorum srv157ilo152,srv158ilo151,srv161ilo150 (age 8d) mgr: srv157ilo152(active, since 24h), standbys: srv161ilo150, srv158ilo151 osd: 15 osds: 15 up (since 8d), 15 in (since 3w) rgw: 3 daemons active (radosgw.srv157ilo152, radosgw.srv158ilo151, radosgw.srv161ilo150) everything seems correct following the (excellent) documentation but i miss something on RGW dashboard configuration : radosgw-admin user create --uid=111 --display-name=cephrgwdash --system gave me an access_key and a secret_key. So I created 2 files rgw_acces and rgw_secret on my /etc/pve/priv folder and paste keys on it next is the error : ceph dashboard set-rgw-api-access-key -i rgw-acces Invalid command: missing required parameter value(<string>) dashboard set-rgw-api-access-key <value> : Set the RGW_API_ACCESS_KEY option value Error EINVAL: invalid command sorry if I made beginner mistake, I'm not used to CEPH and more than that I'm not the devops guy but the hardware / virtualization one tryin to improve its skills. ;) regards, Virttom007

1 0

osd_max_backfills = 1 for one OSD
by Dave Hall 16 Mar '21

16 Mar '21

Hello, I've been trying to get an OSD ready for removal for about a week. RIght now I have 2 pgs backfilling and 21 in backfill_wait. Looking around I found reference to the following command (followed by my output). # ceph config dump | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" global advanced osd_max_backfills 10 osd advanced osd_max_backfills 20 osd advanced osd_recovery_max_active 20 osd.1 advanced osd_max_backfills 1 I'm not sure why osd.1 is special. I'd like to know how to get rid of this. If I increase this value for osd.1 it will likely speed up my backfills, but the special setting for osd.1 will persist. There must be a way to unset this so the general setting is applied, right? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

Inactive pg, how to make it active / or delete
by Szabo, Istvan (Agoda) 16 Mar '21

16 Mar '21

Hi, I have 4 inactive pg in my cluster, the osds are dies where it was before. How can I make it work again? Maybe just threw away because last backfill=max? Based on the pg query it is totally up on other osds. It is an EC 3+1. This is an example inactive pg: ceph pg 44.1f0 query { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "incomplete", "epoch": 5541839, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "info": { "pgid": "44.1f0s0", "last_update": "4863820'2109288", "last_complete": "4863820'2109288", "log_tail": "3881944'2103139", "last_user_version": 11189093, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "4863820'2109288", "reported_seq": "12355046", "reported_epoch": "5541839", "state": "incomplete", "last_fresh": "2021-03-16T10:39:21.058569+0100", "last_change": "2021-03-16T10:39:21.058569+0100", "last_active": "2021-01-20T01:07:18.246158+0100", "last_peered": "2021-01-20T01:07:13.931842+0100", "last_clean": "2021-01-20T01:07:07.392736+0100", "last_became_active": "2021-01-20T01:07:09.187047+0100", "last_became_peered": "2021-01-20T01:07:09.187047+0100", "last_unstale": "2021-03-16T10:39:21.058569+0100", "last_undegraded": "2021-03-16T10:39:21.058569+0100", "last_fullsized": "2021-03-16T10:39:21.058569+0100", "mapping_epoch": 5541572, "log_start": "3881944'2103139", "ondisk_log_start": "3881944'2103139", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 9, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 6149, "ondisk_log_size": 6149, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 356705195545, "num_objects": 98594, "num_object_clones": 758, "num_object_copies": 394376, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 98594, "num_whiteouts": 750, "num_read": 30767, "num_read_kb": 37109881, "num_write": 31023, "num_write_kb": 66655410, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 1000136, "num_bytes_recovered": 3631523354253, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 68 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 4865266, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "34(1)", "pgid": "44.1f0s1", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "119", "reported_epoch": "5520802", "state": "down", "last_fresh": "2021-03-14T07:03:14.383913+0100", "last_change": "2021-03-14T07:03:14.383913+0100", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "2021-03-14T07:03:14.383913+0100", "last_undegraded": "2021-03-14T07:03:14.383913+0100", "last_fullsized": "2021-03-14T07:03:14.383913+0100", "mapping_epoch": 5541572, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 46, 68 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 1, "dne": 0, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "54(3)", "pgid": "44.1f0s3", "last_update": "4863820'2109288", "last_complete": "4863820'2109288", "log_tail": "3881944'2103139", "last_user_version": 11189093, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "4863820'2109288", "reported_seq": "11820744", "reported_epoch": "5336744", "state": "down", "last_fresh": "2021-03-05T16:47:09.479395+0100", "last_change": "2021-03-05T16:45:14.475314+0100", "last_active": "2021-02-15T11:57:07.430181+0100", "last_peered": "2021-01-19T23:23:39.318842+0100", "last_clean": "2021-01-19T23:23:39.318842+0100", "last_became_active": "2021-01-18T23:18:46.888782+0100", "last_became_peered": "2021-01-18T23:18:46.888782+0100", "last_unstale": "2021-03-05T16:47:09.479395+0100", "last_undegraded": "2021-03-05T16:47:09.479395+0100", "last_fullsized": "2021-03-05T16:47:09.479395+0100", "mapping_epoch": 5541572, "log_start": "3881944'2103139", "ondisk_log_start": "3881944'2103139", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 9, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 6149, "ondisk_log_size": 6149, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 356705195545, "num_objects": 98594, "num_object_clones": 758, "num_object_copies": 394376, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 98594, "num_whiteouts": 750, "num_read": 30767, "num_read_kb": 37109881, "num_write": 31023, "num_write_kb": 66655410, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 1000136, "num_bytes_recovered": 3631523354253, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 34, 46, 68, 74, 131 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 4865007, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "62(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "72(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "130", "reported_epoch": "5520914", "state": "down", "last_fresh": "2021-03-14T07:09:19.787519+0100", "last_change": "2021-03-14T07:03:21.216689+0100", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "2021-03-14T07:09:19.787519+0100", "last_undegraded": "2021-03-14T07:09:19.787519+0100", "last_fullsized": "2021-03-14T07:09:19.787519+0100", "mapping_epoch": 5541572, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 34, 46, 68 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 1, "dne": 0, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "74(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "74(3)", "pgid": "44.1f0s3", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "131(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Peering/Incomplete", "enter_time": "2021-03-16T10:39:21.058568+0100", "comment": "not enough complete instances of this PG" }, { "name": "Started/Primary/Peering", "enter_time": "2021-03-16T10:39:21.058483+0100", "past_intervals": [ { "first": "4865006", "last": "5541571", "all_participants": [ { "osd": 33, "shard": 1 }, { "osd": 34, "shard": 1 }, { "osd": 46, "shard": 0 }, { "osd": 54, "shard": 3 }, { "osd": 68, "shard": 2 }, { "osd": 72, "shard": 2 }, { "osd": 74, "shard": 2 }, { "osd": 131, "shard": 2 } ], "intervals": [ { "first": "4865265", "last": "4865269", "acting": "33(1),46(0),68(2)" }, { "first": "4865496", "last": "4865498", "acting": "33(1),46(0),54(3)" }, { "first": "5085650", "last": "5085653", "acting": "34(1),54(3),68(2)" }, { "first": "5114555", "last": "5114581", "acting": "34(1),46(0),68(2)" }, { "first": "5132241", "last": "5132272", "acting": "34(1),46(0),54(3)" }, { "first": "5236601", "last": "5236617", "acting": "46(0),54(3),68(2)" }, { "first": "5520781", "last": "5520798", "acting": "34(1),54(3),72(2)" }, { "first": "5520915", "last": "5520919", "acting": "46(0),54(3),72(2)" }, { "first": "5520920", "last": "5541571", "acting": "34(1),46(0),54(3),72(2)" } ] } ], "probing_osds": [ "34(1)", "46(0)", "54(3)", "62(2)", "72(2)", "74(2)", "74(3)", "131(2)" ], "down_osds_we_would_probe": [ 33, 68 ], "peering_blocked_by": [], "peering_blocked_by_detail": [ { "detail": "peering_blocked_by_history_les_bound" } ] }, { "name": "Started", "enter_time": "2021-03-16T10:39:21.058461+0100" } ], "agent_state": {} } Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Unhealthy Cluster | Remove / Purge duplicate osds | Fix daemon
by Oliver Weinmann 16 Mar '21

16 Mar '21

Hi, On my 3 node Octopus 15.2.5 test cluster, that I haven't used for quite a while, I noticed that it shows some errors: [root@gedasvl02 ~]# ceph health detail INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 HEALTH_WARN 2 failed cephadm daemon(s) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.0 on gedaopl02 is in error state daemon node-exporter.gedaopl01 on gedaopl01 is in error state The error about the osd.0 is strange since osd.0 is actually up and running but on a different node. I guess I missed to correctly remove it from node gedaopl02 and then added a new osd to a different node gedaopl01 and now there are duplicate osd ids for osd.0 and osd.2. [root@gedasvl02 ~]# ceph orch ps INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID alertmanager.gedasvl02 gedasvl02 running (6h) 7m ago 4M 0.20.0 docker.io/prom/alertmanager:v0.20.0 0881eb8f169f 5b80fb977a5f crash.gedaopl01 gedaopl01 stopped 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 810cf432b6d6 crash.gedaopl02 gedaopl02 running (5h) 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 34ab264fd5ed crash.gedaopl03 gedaopl03 running (2d) 7m ago 2d 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 233f30086d2d crash.gedasvl02 gedasvl02 running (6h) 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 ea3d3e7c4f58 grafana.gedasvl02 gedasvl02 running (6h) 7m ago 4M 6.6.2 docker.io/ceph/ceph-grafana:6.6.2 a0dce381714a 5a94f3e41c32 mds.cephfs.gedaopl01.zjuhem gedaopl01 stopped 7m ago 3M <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mds.cephfs.gedasvl02.xsjtpi gedasvl02 running (6h) 7m ago 3M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 26e7c8759d89 mgr.gedaopl03.zilwbl gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 e18b6f40871c mon.gedaopl03 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 5afdf40e41ba mon.gedasvl02 gedasvl02 running (6h) 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 e83dfcd864aa node-exporter.gedaopl01 gedaopl01 error 7m ago 4M 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 0fefcfcc9639 node-exporter.gedaopl02 gedaopl02 running (5h) 7m ago 4M 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf f459045b7e41 node-exporter.gedaopl03 gedaopl03 running (2d) 7m ago 2d 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 3bd9f8dd6d5b node-exporter.gedasvl02 gedasvl02 running (6h) 7m ago 4M 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 72e96963261e *osd.0 gedaopl01 running (5h) 7m ago 5h 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 ed76fafb1988** **osd.0 gedaopl02 error 7m ago 4M <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown>* osd.1 gedaopl01 running (4h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 41a43733e601 *osd.2 gedaopl01 stopped 7m ago 4M <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown>** **osd.2 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 ac9e660db2fb* osd.3 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 bde17b5bb2fb osd.4 gedaopl02 running (5h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 7cc3ef7c4469 osd.5 gedaopl02 running (5h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 761b96d235e4 osd.6 gedaopl02 running (5h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 d047b28fe2bd osd.7 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 3b54b01841f4 osd.8 gedaopl01 running (5h) 7m ago 5h 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 cdd308cdc82b prometheus.gedasvl02 gedasvl02 running (5h) 7m ago 4M 2.18.1 docker.io/prom/prometheus:v2.18.1 de242295e225 591cef3bbaa4 Is there a way to clean / purge the stopped and error ones? I don't know what is wrong with the node-exporter. Because looking at podman ps -a on gedaopl01 looks ok. Maybe also a zombie daemon? [root@gedaopl01 ~]# podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e71898f7d038 docker.io/prom/node-exporter:v0.18.1 --no-collector.ti... 54 seconds ago Up 54 seconds ago ceph-d0920c36-2368-11eb-a5de-005056b703af-node-exporter.gedaopl01 41a43733e601 docker.io/ceph/ceph:v15 -n osd.1 -f --set... 5 hours ago Up 5 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.1 810cf432b6d6 docker.io/ceph/ceph:v15 -n client.crash.g... 6 hours ago Up 6 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-crash.gedaopl01 cdd308cdc82b docker.io/ceph/ceph:v15 -n osd.8 -f --set... 6 hours ago Up 6 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.8 ed76fafb1988 docker.io/ceph/ceph:v15 -n osd.0 -f --set... 6 hours ago Up 6 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0 I replaced the very old disks with some brand new SAMSUMG PM883 and would like to upgrade to 15.2.9. But the upgrade guide recommends to do this on a healthy cluster only. :) Cheers, Oliver

2 3

Safe to remove osd or not? Which statement is correct?
by Szabo, Istvan (Agoda) 16 Mar '21

16 Mar '21

Hi Gents, There is a cluster with 14 hosts in this state: https://i.ibb.co/HPF3Pdr/6-ACB2-C5-B-6-B54-476-B-835-D-227-E9-BFB1247.jpg There is a host based crush rule ec 3:1 and there are 3 hosts where are osds down. Unfortunately there are pools with 3 replicas also which is host based. 2 hosts have 2 osds down,1 host has 1 osd down, which means I guess if we don’t bring it back, 0.441% data loss is going to happen. Am I right? I hope I’m not right because there isn’t any missplaced objects, only degraded. The problem is that the osds are not starting so somehow these 5 osds should be removed, but I’m curious which is my statement is correct. Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

millions slow ops on a cluster without load
by Szabo, Istvan (Agoda) 16 Mar '21

16 Mar '21

We have a cluster with a huge amount of warnings like this even if nothing is going on in the cluster. It makes mgr physical memory full, mon db maxed out 5 osds can't start :/ [WRN] slow request osd_op(mds.0.537792:26453 43.38 43:1d6c5587:::10000fe56a6.00000000:head [create,setxattr parent (367) in=373b,setxattr layout (30) in=36b] snapc 0=[] RETRY=131 ondisk+retry+write+known_if_redirected+full_force e5532574) initiated 2021-03-15T16:44:18.428585+0100 currently delayed Is there a way to somehow stop it? MDS service stopped also. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

v14.2.18 Nautilus released
by David Galloway 15 Mar '21

15 Mar '21

We're happy to announce the 18th backport release in the Nautilus series. It fixes a regression introduced in 14.2.17 in which the manager module tries to use a couple python modules that do not exist in some environments. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-18-nautilus-released Notable Changes --------------- * This release fixes issues loading the dashboard and volumes manager modules in some environments. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.18.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: befbc92f3c11eedd8626487211d200c0b44786d9

1 0

Best way to add OSDs - whole node or one by one?
by Dave Hall 15 Mar '21

15 Mar '21

Hello, I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB. Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%. My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes? Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

4 5

MDS pinning: ceph.dir.pin: No such attribute
by Frank Schilder 15 Mar '21

15 Mar '21

Hi, I seem to have a problem with the extended attributes for MDS pinning. Ceph version is mimic-13.2.10 and the documentation (https://web.archive.org/web/20190716110503/http://docs.ceph.com/docs/mimic/…) says I should be able to setfattr -n ceph.dir.pin -v 2 path/to/dir When I try this, I get the following: # setfattr -n ceph.dir.pin -v 1 . # getfattr -n ceph.dir.pin . .: ceph.dir.pin: No such attribute Is this expected? How can I check that pinning is in effect as intended? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 5

Current BlueStore cache autotune (memory target) is respect media?
by Konstantin Shalygin 15 Mar '21

15 Mar '21

Hi, Current default's (for nautilus, for example) is respect media type (hdd/ssd/hybrid) or for all OSD's current memory_target is 4GiB? Thanks, k

2 2

lvm fix for reseated reseated device
by Dan van der Ster 15 Mar '21

15 Mar '21

Hi all, Occasionally we see a bus glitch which causes a device to disappear then reappear with a new /dev/sd name. This crashes the osd (giving IO errors) but after a reboot the OSD will be perfectly fine. We're looking for a way to reeactivate osd like this without rebooting. For example, logs showing sdd disappear then reappear as sdq from this morning are in the P.S. We tried pvscan, vgscan, lvscan, but in all cases when trying to activate the osd we get an I/O error, as if the dm entry for the lv/vg is still referring to /dev/sdd. Is there some obvious way to properly tear down what refers to sdd / 0:0:3:0 so that we can activate sdq ? (In this case, we have already rebooted the box so I won't be able to test immediately.) Best Regards, Dan Mar 15 04:57:36 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: device_block, handle(0x001c) Mar 15 04:57:38 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: device_unblock and setting to running, handle(0x001c) Mar 15 04:57:38 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: [sdd] Synchronizing SCSI cache Mar 15 04:57:38 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Stopping LVM event activation on device 8:48... Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x300062b2038af0c3) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: removing handle(0x001c), sas_addr(0x300062b2038af0c3) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: enclosure logical id(0x500062b2038af0c0), slot(1) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch lvm[1157119]: pvscan[1157119] device 8:48 not found. Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: lvm2-pvscan@8:48.service: Succeeded. Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Stopped LVM event activation on device 8:48. ... Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: Direct-Access ATA Micron_5200_MTFD U020 PQ: 0 ANSI: 6 Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: SATA: handle(0x001c), sas_addr(0x300062b2038af0c3), phy(3), device_name(0x0000000000000000) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: enclosure logical id (0x500062b2038af0c0), slot(1) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: enclosure level(0x0000), connector name( ) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: Power-on or device reset occurred Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: Attached scsi generic sg3 type 0 Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: end_device-0:16: add: handle(0x001c), sas_addr(0x300062b2038af0c3) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] 1875385008 512-byte logical blocks: (960 GB/894 GiB) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] 4096-byte physical blocks Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] Write Protect is off Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] Attached SCSI disk Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Starting LVM event activation on device 65:0... Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: pvscan[1157327] PV /dev/sdq online, VG ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5 is complete. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: pvscan[1157327] VG ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5 run autoactivation. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: PVID qlETw1-KssL-Vc9P-MpAE-ED7I-gnWS-kO4DWF read from /dev/sdq last written to /dev/sdd. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: pvscan[1157327] VG ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5 not using quick activation. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: 1 logical volume(s) in volume group "ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5" now active Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Started LVM event activation on device 65:0.

3 3

Alertmanager not using custom configuration template
by Marc 'risson' Schmitt 15 Mar '21

15 Mar '21

Hi, I'm trying to use a custom template for Alertmanager deployed with Cephadm. Following its documentation[1], I set the option `mgr/cephadm/alertmanager_alertmanager.yml` to my own template, restarted the mgr, and re-deployed Alertmanager. However, Cephadm seems to always use its internal template. After some debugging, I found that the mgr indeed queries for `mgr/cephadm/alertmanager_alertmanager.yml`, but the end template always is the default one. Am I doing something wrong? Is anyone else having this issue? [1] https://docs.ceph.com/en/octopus/cephadm/monitoring/#using-custom-configura… Regards, -- Marc 'risson' Schmitt CRI - EPITA

2 4

MDS stuck in replay/resolve stats
by Zhenshi Zhou 15 Mar '21

15 Mar '21

Hi, I encountered an issue lately. I have a cephfs cluster on 14.2.11 with 5 active MDS and 5 stand-replay MDS. Metadata pool is on SSD and datapool is on SATA. 2 of MDS restart frequently and the replay MDS stuck into replay and resolve states and never active. What's wrong with my MDS? *the restart MDS logs:* -214> 2021-03-15 17:39:00.898 7fafe3456700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -213> 2021-03-15 17:39:00.898 7fafe3456700 0 mds.beacon.aiceph-08 Skipping beacon heartbeat to monitors (last acked 54.511s ago); MDS internal heartbeat is not healthy! -212> 2021-03-15 17:39:00.911 7fafdf44e700 5 mds.3.migrator export_finish [dir 0x2000e252701.010100000110* /some/dirs/ [2,head] auth{2=2} v=143307 cv=0/0 dir_auth=2,3 state= 1073742850|frozentree|exporting f(v3086 m2021-03-12 04:21:43.383480 1303=1303+0) n(v3128 rc2021-03-12 04:21:43.384480 b119066284 1303=1303+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 frozen=1 subtree=1 importing=0 replicated=1 waiter=0 authpin=0 tempexporting=1 0x55df67f36f00] ...... -204> 2021-03-15 17:39:01.398 7fafe3456700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -203> 2021-03-15 17:39:01.398 7fafe3456700 0 mds.beacon.aiceph-08 Skipping beacon heartbeat to monitors (last acked 55.011s ago); MDS internal heartbeat is not healthy! -202> 2021-03-15 17:39:01.401 7fafe5c5b700 1 mds.beacon.aiceph-08 MDS connection to Monitors appears to be laggy; 55.014s since last acked beacon -201> 2021-03-15 17:39:01.401 7fafe5c5b700 5 mds.3.78900 laggy, deferring export_prep(0x2000e252701.110110011001*) v1 -200> 2021-03-15 17:39:01.402 7fafe3c57700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 -199> 2021-03-15 17:39:01.402 7fafe3c57700 1 mds.3.78900 skipping upkeep work because connection to Monitors appears laggy -198> 2021-03-15 17:39:01.402 7fafe5c5b700 1 mds.3.objecter ms_handle_reset 0x55dead0ba800 session 0x55dd8944e160 osd.24 -197> 2021-03-15 17:39:01.402 7fafe5c5b700 4 mgrc ms_handle_reset ms_handle_reset con 0x55e09c44b400 -196> 2021-03-15 17:39:01.402 7fafe5c5b700 4 mgrc reconnect Terminating session with v2:172.25.64.235:6800/35103 -195> 2021-03-15 17:39:01.402 7fafe8eea700 10 monclient: get_auth_request con 0x55e15190f000 auth_method 0 -194> 2021-03-15 17:39:01.402 7fafe5c5b700 4 mgrc reconnect Starting new session with [v2:172.25.64.235:6800/35103,v1:172.25.64.235:6801/35103] -193> 2021-03-15 17:39:01.402 7fafe86e9700 10 monclient: get_auth_request con 0x55dead0ba800 auth_method 0 -192> 2021-03-15 17:39:01.467 7fafe1452700 2 mds.3.cache Memory usage: total 60577760, rss 7823000, heap 315644, baseline 315644, 28365 / 2610243 inodes have caps, 33114 caps, 0.0126862 caps per inode -191> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mds.3.78900 handle_osd_map epoch 12458, 2 new blacklist entries -190> 2021-03-15 17:39:01.467 7fafe5c5b700 10 monclient: _renew_subs -189> 2021-03-15 17:39:01.467 7fafe5c5b700 10 monclient: _send_mon_message to mon.aiceph-03 at v2:172.25.72.42:3300/0 -188> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mgrc ms_handle_reset ms_handle_reset con 0x55dead0ba800 -187> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mgrc reconnect Terminating session with v2:172.25.64.235:6800/35103 -186> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mgrc reconnect waiting to retry connect until 2021-03-15 17:39:02.403432 -185> 2021-03-15 17:39:01.467 7fafdf44e700 5 mds.3.migrator export_finish [dir 0x2000e252701.010100001110* /ai_root/NLP/Member/zhangpengshen/mode_repo/phoneme_recognizer/wav2letter_simple/4000h_npy/npy/ [2,head] auth{2=2} v=142958 cv=0/0 dir_auth=2,3 state= 1073742850|frozentree|exporting f(v3086 m2021-03-12 04:21:26.970694 1333=1333+0) n(v3128 rc2021-03-12 04:21:26.970694 b121920160 1333=1333+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 frozen=1 subtree=1 importing=0 replicated=1 waiter=0 authpin=0 tempexporting=1 0 x55df671ebe00] ...... -177> 2021-03-15 17:39:01.898 7fafe3456700 5 mds.beacon.aiceph-08 Sending beacon up:active seq 1364089 -176> 2021-03-15 17:39:01.898 7fafe3456700 10 monclient: _send_mon_message to mon.aiceph-03 at v2:172.25.72.42:3300/0 ...... -3> 2021-03-15 17:39:11.669 7fafdf44e700 5 mds.3.migrator export_finish [dir 0x2000e252701.010111101010* /some/dir/ [2,head] auth{2=2} v=143028 cv=143028/143028 dir_auth=2,3 state=1073742850|frozentree|exporting f(v3086 m2021-03-12 04:22:17.528035 1340=1340+0) n(v3128 rc2021-03-12 04:22:17.529035 b125945332 1340=1340+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 frozen=1 subtree=1 importing=0 replicated=1 waiter=0 authpin=0 tempexporting=1 0x55e029a94500] -2> 2021-03-15 17:39:11.728 7fafe5c5b700 1 mds.aiceph-08 Updating MDS map to version 199401 from mon.2 -1> 2021-03-15 17:39:11.728 7fafe5c5b700 1 mds.aiceph-08 Map removed me [mds.aiceph-08{3:977486} state up:active seq 1364038 export targets 0,1,2 addr [v2:172.25.72.47:6960/1073018517,v1:172.25.72.47:6961/1073018517]] from cluster; respawning! See cluster/monitor logs for details. 0> 2021-03-15 17:39:11.728 7fafe5c5b700 1 mds.aiceph-08 respawn! *the resolve MDS logs:* 2021-03-15 17:39:12.427 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199401 from mon.0 2021-03-15 17:39:41.092 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199402 from mon.0 2021-03-15 17:39:41.111 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199403 from mon.0 2021-03-15 17:39:41.111 7f6c0b139700 1 mds.2.199403 handle_mds_map i am now mds.2.199403 2021-03-15 17:39:41.111 7f6c0b139700 1 mds.2.199403 handle_mds_map state change up:standby-replay --> up:replay 2021-03-15 17:39:41.715 7f6c0492c700 1 mds.2.199403 standby_replay_restart (final takeover pass) 2021-03-15 17:39:41.715 7f6c0492c700 1 mds.2.199403 opening purge_queue (async) 2021-03-15 17:39:41.715 7f6c0492c700 1 mds.2.199403 opening open_file_table (async) 2021-03-15 17:39:41.716 7f6c0492c700 1 mds.2.199403 Finished replaying journal 2021-03-15 17:39:41.716 7f6c0492c700 1 mds.2.199403 making mds journal writeable 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199404 from mon.0 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 handle_mds_map i am now mds.2.199403 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 handle_mds_map state change up:replay --> up:resolve 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 resolve_start 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 reopen_log 2021-03-15 17:40:53.758 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199407 from mon.0 2021-03-15 17:41:09.478 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199408 from mon.0 2021-03-15 17:42:05.671 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199409 from mon.0 2021-03-15 17:42:15.704 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199410 from mon.0 *the replay MDS logs:* 2021-03-15 17:37:55.079 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199400 from mon.0 2021-03-15 17:38:52.763 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199401 from mon.0 2021-03-15 17:38:52.763 7f7a6df1c700 1 mds.3.199401 handle_mds_map i am now mds.3.199401 2021-03-15 17:38:52.763 7f7a6df1c700 1 mds.3.199401 handle_mds_map state change up:standby-replay --> up:replay 2021-03-15 17:39:21.428 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199402 from mon.0 2021-03-15 17:39:21.447 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199403 from mon.0 2021-03-15 17:39:22.452 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199404 from mon.0 2021-03-15 17:40:34.093 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199407 from mon.0 2021-03-15 17:40:49.813 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199408 from mon.0 2021-03-15 17:41:46.006 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199409 from mon.0 2021-03-15 17:41:56.039 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199410 from mon.0

1 0

Re: How big an OSD disk could be?
by Janne Johansson 15 Mar '21

15 Mar '21

Den fre 12 mars 2021 kl 18:10 skrev huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn>: > Dear cephers, > Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? > Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit? I think your patience will be the limit. If you stick 8x or 10x 20TB drives in a box, it will take a very long time to fill them to say, 50%. That would be some 80-100TB to move over the network and a lot of IO spread over only 8-10 drives before it settles, which would be a long time on spinning drives. Unless the objects are huge, 100TB of files/objects would mean many many millions of writes just to fill the OSD and its list of contents. For spinners, it would mean many many millions of head movements just to write it all out, hopefully never encountering a random bad block. After you have filled that up, if such a host crashes or needs maintenance, another 80-100TB will need recreating from the other huge drives. I don't think there are specific limitations on the size itself, but as the single drive becomes larger and larger, just adding a new host or a drive will mean the cluster is rebalancing for days or weeks if not more. At some point you would end up having the cluster almost never in HEALTH_OK state because of normal replacements, expansions and other surprises, which in turn could cause secondary problems with mon DBs and things like that. -- May the most significant bit of your life be positive.

3 2

How big an OSD disk could be?
by huxiaoyu＠horebdata.cn 14 Mar '21

14 Mar '21

Dear cephers, Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit? best regards, samuel huxiaoyu(a)horebdata.cn

7 13

Re: Some confusion around PG, OSD and balancing issue
by Darrin Hodges 14 Mar '21

14 Mar '21

HI Matthew, the results of the commands are: ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 190 TiB 61 TiB 129 TiB 129 TiB 67.70 TOTAL 190 TiB 61 TiB 129 TiB 129 TiB 67.70 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 1 31 MiB 0 B 31 MiB 73 92 MiB 0 B 92 MiB 0 9.8 TiB N/A N/A 73 0 B 0 B libvirt-pool 3 512 43 TiB 43 TiB 9.1 MiB 11.18M 128 TiB 128 TiB 27 MiB 81.21 9.8 TiB N/A N/A 11.18M 0 B 0 B ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 32288 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth pool 3 'libvirt-pool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 31905 lfor 0/28367/30002 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.003773", "last_optimize_started": "Sun Mar 14 06:28:22 2021", "mode": "upmap", "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } We are using a vm with rbd disk, in this case 6 x 10T disk, given that the pool is replicated 3, it going to run out of space. We are in the process of upgrading the 3T disks to 8T but still won't be enough space if I understand things correctly, may need to considering changing to replication2, is that possible? many thanks Darrin On 13/3/21 3:07 pm, Matthew H wrote: > Hey Darrin, > > Can you provide the output of the following commands? > > ceph df detail > ceph osd pool ls detail > ceph balancer status > > Thanks so much, > > ------------------------------------------------------------------------ > *From:* Darrin Hodges <darrin(a)catalyst-au.net> > *Sent:* Wednesday, March 10, 2021 8:41 PM > *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Some confusion around PG, OSD and balancing issue > > HI all, > > Just looking for clarification around the relationship between PGs,OSDs > and balancing on a ceph (octopus) cluster. We have pg autobalance on > and balancing is set to upmap. There are 2 pools, one is the default > metric pool with 1 pg, the other is the pool we are using for > everything, it has 512 PG. There are 60 OSD's split across 4 hosts. The > USD usage ranges between 39% and 68%: > > > * current cluster score 0.048908 > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META AVAIL %USE VAR PGS STATUS > 0 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.3 > GiB 1.5 TiB 47.08 0.86 19 up > > 1 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 3.2 MiB 2.3 > GiB 1.4 TiB 49.46 0.90 20 up > > 2 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.8 MiB 2.4 > GiB 1.3 TiB 51.81 0.95 21 up > > 3 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 589 KiB 2.4 > GiB 1.3 TiB 51.73 0.95 21 up > > 4 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.2 MiB 2.7 > GiB 1.1 TiB 59.05 1.08 24 up > > 5 hdd 9.00000 1.00000 9.1 TiB 5.2 TiB 5.2 TiB 6.9 MiB 8.2 > GiB 3.9 TiB 56.75 1.04 77 up > > 6 hdd 2.74550 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.9 MiB 2.8 > GiB 1.3 TiB 54.21 0.99 22 up > > 7 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.7 > GiB 1.4 TiB 49.32 0.90 20 up > > 8 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.3 > GiB 1.4 TiB 49.36 0.90 20 up > > 9 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 3.3 MiB 3.0 > GiB 1.1 TiB 59.21 1.08 24 up > > 10 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.2 MiB 3.4 > GiB 941 GiB 66.53 1.22 27 up > > 11 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.9 MiB 2.0 > GiB 1.7 TiB 39.69 0.73 16 up > > 12 hdd 2.74550 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 1.2 MiB 2.1 > GiB 1.5 TiB 44.69 0.82 18 up > > 13 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 2.4 MiB 1.9 > GiB 1.7 TiB 39.59 0.72 16 up > > 14 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.1 MiB 3.1 > GiB 945 GiB 66.37 1.21 27 up > > 15 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 1.7 MiB 2.6 > GiB 1.2 TiB 56.68 1.04 23 up > > 16 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 3.3 MiB 2.3 > GiB 1.5 TiB 46.76 0.86 19 up > > 17 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 2.8 MiB 3.1 > GiB 953 GiB 66.12 1.21 26 up > > 18 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 > GiB 1010 GiB 64.08 1.17 26 up > > 19 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 3.0 MiB 2.9 > GiB 1016 GiB 63.88 1.17 26 up > > 20 hdd 9.00000 1.00000 9.1 TiB 4.7 TiB 4.7 TiB 28 MiB 7.5 > GiB 4.4 TiB 51.47 0.94 70 up > > 21 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 2.8 MiB 2.2 > GiB 1.5 TiB 44.43 0.81 18 up > > 22 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 653 KiB 2.4 > GiB 1.3 TiB 51.89 0.95 21 up > > 23 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 > GiB 1.1 TiB 59.26 1.08 24 up > > 24 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 3.1 MiB 3.0 > GiB 1.2 TiB 56.67 1.04 23 up > > 25 hdd 2.74599 0.90002 2.7 TiB 1.7 TiB 1.7 TiB 5.3 MiB 4.0 > GiB 1.1 TiB 61.47 1.12 25 up > > 26 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 1.5 MiB 2.4 > GiB 1.3 TiB 51.82 0.95 21 up > > 27 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.1 MiB 2.2 > GiB 1.5 TiB 46.93 0.86 19 up > > 28 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.6 MiB 2.2 > GiB 1.5 TiB 46.88 0.86 19 up > > 29 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 557 KiB 2.6 > GiB 1.4 TiB 49.37 0.90 20 up > > 45 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 > GiB 1.1 TiB 59.18 1.08 24 up > > 46 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 28 MiB 2.5 > GiB 1.3 TiB 51.76 0.95 22 up > > 47 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 977 KiB 3.1 > GiB 1.1 TiB 59.07 1.08 24 up > > 48 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 625 KiB 2.6 > GiB 1.5 TiB 46.86 0.86 19 up > > 49 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.9 MiB 3.2 > GiB 1.1 TiB 61.68 1.13 25 up > > 50 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.0 MiB 2.7 > GiB 1.1 TiB 61.53 1.13 25 up > > 51 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.1 TiB 2.5 MiB 2.3 > GiB 1.6 TiB 41.88 0.77 17 up > > 52 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 1.8 MiB 3.8 > GiB 942 GiB 66.49 1.22 27 up > > 53 hdd 9.00000 1.00000 9.1 TiB 4.3 TiB 4.3 TiB 6.2 MiB 6.8 > GiB 4.8 TiB 47.46 0.87 64 up > > 54 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 > GiB 1008 GiB 64.13 1.17 26 up > > 55 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 > GiB 1014 GiB 63.95 1.17 26 up > > 56 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.8 > GiB 1013 GiB 63.99 1.17 26 up > > 57 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 2.9 MiB 2.6 > GiB 1.2 TiB 56.67 1.04 23 up > > 58 hdd 2.74599 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.2 MiB 2.5 > GiB 1.3 TiB 54.41 1.00 22 up > > 59 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 > GiB 1015 GiB 63.92 1.17 26 up > > 30 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 1.6 MiB 3.1 > GiB 878 GiB 68.79 1.26 28 up > > 31 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.2 > GiB 1.5 TiB 47.03 0.86 19 up > > 32 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 3.2 MiB 3.3 > GiB 944 GiB 66.45 1.22 27 up > > 33 hdd 2.74599 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.1 MiB 1.9 > GiB 1.7 TiB 39.88 0.73 16 up > > 34 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 28 MiB 3.0 > GiB 874 GiB 68.93 1.26 29 up > > 35 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.3 MiB 2.2 > GiB 1.5 TiB 47.01 0.86 19 up > > 36 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 1.6 MiB 3.0 > GiB 947 GiB 66.31 1.21 27 up > > 37 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 1.8 MiB 2.9 > GiB 1.1 TiB 58.96 1.08 24 up > > 38 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 2.8 MiB 3.4 > GiB 877 GiB 68.82 1.26 28 up > > 39 hdd 2.74599 0.85004 2.7 TiB 1.6 TiB 1.6 TiB 2.4 MiB 3.5 > GiB 1.1 TiB 59.20 1.08 24 up > > 40 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.5 MiB 2.3 > GiB 1.4 TiB 49.35 0.90 20 up > > 41 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.3 MiB 2.4 > GiB 1.3 TiB 51.73 0.95 21 up > > 42 hdd 9.00000 1.00000 9.1 TiB 4.5 TiB 4.5 TiB 2.1 MiB 7.2 > GiB 4.6 TiB 49.76 0.91 67 up > > 43 hdd 2.74599 0.90002 2.7 TiB 1.6 TiB 1.6 TiB 2.9 MiB 3.2 > GiB 1.1 TiB 59.17 1.08 24 up > > 44 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 4.0 MiB 2.2 > GiB 1.5 TiB 44.34 0.81 18 up > > > > 54 of the OSD are 3TiB and there are four 8TiB - can this cause > imbalance? Or do I need to increase the PG from 512? or is this as good > as it gets? > > > > many thanks > > Darrin > > -- > CONFIDENTIALITY NOTICE: This email is intended for the named > recipients only. It may contain privileged, confidential or copyright > information. If you are not the named recipients, any use, reliance > upon, disclosure or copying of this email or any attachments is > unauthorised. If you have received this email in error, please reply > via email or telephone +61 2 8004 5928. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

1 0

Removing secondary data pool from mds
by Michael Thomas 13 Mar '21

13 Mar '21

I have a cephfs secondary (non-root) data pool with unfound and degraded objects that I have not been able to recover[1]. I created an additional data pool and used "setfattr -n ceph.dir.layout.pool' and a very long rsync to move the files off of the degraded pool and onto the new pool. This has completed, and using find + 'getfattr -n ceph.file.layout.pool', I verified that no files are using the old pool anymore. No ceph.dir.layout.pool attributes point to the old pool either. However, the old pool still reports that there are objects in the old pool, likely the same ones that were unfound/degraded from before: https://pastebin.com/qzVA7eZr Based on a old message from the mailing list[2], I checked the MDS for stray objects (ceph daemon mds.ceph4 dump cache file.txt ; grep -i stray file.txt) and found 36 stray entries in the cache: https://pastebin.com/MHkpw3DV. However, I'm not certain how to map these stray cache objects to clients that may be accessing them. 'rados -p fs.data.archive.frames ls' shows 145 objects. Looking at the parent of each object shows 2 strays: for obj in $(cat rados.ls.txt) ; do echo $obj ; rados -p fs.data.archive.frames getxattr $obj parent | strings ; done [...] 10000020fa1.00000000 10000020fa1 stray6 10000020fbc.00000000 10000020fbc stray6 [...] ...before getting stuck on one object for over 5 minutes (then I gave up): 1000005b1af.00000083 What can I do to make sure this pool is ready to be safely deleted from cephfs (ceph fs rm_data_pool archive fs.data.archive.frames)? --Mike [1]https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QHFOGEKXK7VDNNSKR74BA6IIMGGIXBXA/#7YQ6SSTESM5LTFVLQK3FSYFW5FDXJ5CF [2]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005233.h…

2 5

Location of Crush Map and CEPH metadata
by Ed Kalk 13 Mar '21

13 Mar '21

Hello, I have been googling for the answer to this and not found it. Does anyone know this? Where does CEPH store the crush map and the critical cluster metadata? What prevents a loss of this metadata when a node is lost? -- Thank you for your time, Edward H. Kalk IV Information Technology Dept. Server Specialist Datacenter Virtualization and Storage Systems Socket Telecom, LLC. 2703 Clark Lane Columbia, MO 65202 573-817-0000 or 800-socket3 X218

3 2

ceph repo cert expired
by Philip Brown 12 Mar '21

12 Mar '21

To whom it may concern: failure: repodata/repomd.xml from Ceph: [Errno 256] No more mirrors to try. https://download.ceph.com/rpm-octopus/el7/x86_64/repodata/repomd.xml: [Errno 14] curl#60 - "Peer's Certificate has expired."

1 0

Ceph 14.2.17 ceph-mgr module issue
by Stefan Kooman 12 Mar '21

12 Mar '21

Hi, After upgrading a Ceph cluster to 14.2.17 with ceph-ansible (docker containers) the manager hits an issue: Module 'volumes' has failed dependency: No module named typing, python trace: 2021-03-12 17:04:22.358 7f299ac75e40 1 mgr[py] Loading python module 'volumes' 2021-03-12 17:04:22.458 7f299ac75e40 -1 mgr[py] Module not found: 'volumes' 2021-03-12 17:04:22.458 7f299ac75e40 -1 mgr[py] Traceback (most recent call last): File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in <module> from .module import Module File "/usr/share/ceph/mgr/volumes/module.py", line 10, in <module> from .fs.volume import VolumeClient File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 13, in <module> from .operations.subvolume import open_subvol, create_subvol, remove_subvol, \ File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 8, in <module> from .versions import loaded_subvolumes File "/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py", line 9, in <module> from .subvolume_v1 import SubvolumeV1 File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v1.py", line 9, in <module> from typing import List, Dict ImportError: No module named typing I created tracker [1] for this issue with this information. Not sure if non containerized deployments hit this issue as well. I will find thta out somewhere next week. FYI, Gr. Stefan [1]: https://tracker.ceph.com/issues/49770

4 7

Question about delayed write IOs, octopus, mixed storage
by Philip Brown 12 Mar '21

12 Mar '21

I'm running some tests with mixed storage units, and octopus. 8 nodes, each with 2 SSDs, and 8 HDDs . the SSDsare relatively small: around 100GB each. Im mapping 8 rbds, striping them together, and running fio on them for testing. # fio --filename=/...../fio.testfile --size=120GB --rw=randrw --bs=8k --direct=1 --ioengine=libaio --iodepth=64 --numjobs=4 --time_based --group_reporting --name=readwritelatency-test-job --runtime=120 --eta-newline=1 Trouble is, I'm seeing sporadic delays of IOs. When I test ZFS, for example, it has this neat wait clumping status check: zpool iostat -w 20 and it shows me that some write io's are taking over 4 secomds to complete. Many are taking 1s or 2s This kind of thing has sort of happened before(but previously, I think I was using SSDs exclusively). When I emailed the list, people suggested turning off RBD cache, which worked great in that situation. This time, I have already done that (I believe), but still see this behavior. Would folks have any further suggestions to smooth performance out? The odd thing is, I read that bluestore is supposed to smooth things out and provide consistent response time, but that doesnt seem to be the case. Sample output from the zpool iostat below: twelve total_wait disk_wait syncq_wait asyncq_wait latency read write read write read write read write scrub trim ---------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- .. snip ... 1ms 1.01K 0 1.00K 0 0 0 0 0 0 0 2ms 29 0 29 18 0 0 0 1 0 0 4ms 23 3 23 14 0 0 0 3 0 0 8ms 64 6 64 9 0 0 0 7 0 0 16ms 74 10 74 59 0 0 0 11 0 0 33ms 24 17 24 154 0 0 0 19 0 0 67ms 7 25 7 100 0 0 0 26 0 0 134ms 3 40 3 36 0 0 0 36 0 0 268ms 1 59 1 18 0 0 0 59 0 0 536ms 0 116 0 3 0 0 0 113 0 0 1s 0 109 0 0 0 0 0 98 0 0 2s 0 24 0 0 0 0 0 20 0 0 4s 0 2 0 0 0 0 0 1 0 0 8s 0 0 0 0 0 0 0 0 0 0 17s 0 0 0 0 0 0 0 0 0 0 -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 6

Container deployment - Ceph-volume activation
by Cloud Guy 12 Mar '21

12 Mar '21

Hello, TL;DR Looking for guidance on ceph-volume lvm activate --all as it would apply to a containerized ceph deployment (Nautilus or Octopus). Detail: I’m planning to upgrade my Nautilus non-container cluster to Octopus (eventually containerized). There’s an expanded procedure that was tested and working in our lab, however won’t go into the whole process. My question is around existing OSD hosts. I have to re-platform the host OS, and one of the ways in the OSDs were reactivated previously when this was done (non-containerized) was to install ceph packages, deploy keys, config, etc. then run ceph-volume lvm activate --all to magically bring up all OSDs. Looking for a similar approach except if the OSDs are containerized, and I re-platform the host OS (Centos -> Ubuntu), how could I reactivate all OSDs as containers and avoid rebuilding data on the OSDs? Thank you.

4 4

OSDs crashing after server reboot.
by Cassiano Pilipavicius 12 Mar '21

12 Mar '21

Hi, please if someone know how to help, I have an HDD pool in mycluster and after rebooting one server, my osds has started to crash. This pool is a backup pool and have OSD as failure domain with an size of 2. After rebooting one server, My osds started to crash, and the thing is only getting worse. I have then tried to run ceph-bluestore-tool repair and I receive what I think is the same error that shows on the osd logs: [root@cwvh13 ~]# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-81 --log-level 10 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f6467ffcec0 time 2021-03-11 12:13:12.121766 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: 53: FAILED ceph_assert(r == 0) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f645e1a7b27] 2: (()+0x25ccef) [0x7f645e1a7cef] 3: (()+0x3cd57f) [0x5642e85c457f] 4: (HybridAllocator::~HybridAllocator()+0x17) [0x5642e85f3f37] 5: (BlueStore::_close_alloc()+0x42) [0x5642e84379d2] 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x5642e84bbac8] 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x293) [0x5642e84bbf13] 8: (main()+0x13cc) [0x5642e83caaec] 9: (__libc_start_main()+0xf5) [0x7f645ae24555] 10: (()+0x1fae9f) [0x5642e83f1e9f]

2 3

Ceph server
by Ignazio Cassano 12 Mar '21

12 Mar '21

Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ? A+ Server 1113S-WN10RT Barebone Supermicro A+ Server 1113S-WN10RT - 1U - 10x U.2 NVMe - 2x M.2 - Dual 10-Gigabit LAN - 750W Redundant Processor AMD EPYC™ 7272 Processor 12-core 2.90GHz 64MB Cache (120W) Memory 8 x 8GB PC4-25600 3200MHz DDR4 ECC RDIMM U.2/U.3 NVMe Drive 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive 2 x 240GB Intel® SSD D3-S4610 Series 2.5" SATA 6.0Gb/s Solid State Drive Network Card 2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+) Server Management Supermicro Update Manager (SUM) (OOB Management Package), included Operating System No Windows Operating System Warranty 3 Year Depot Warranty (Return for Repair) RAID U.2 [NVMe, 10 ports] No RAID (*OS) -> 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Thanks Ignazio

4 9

ceph boostrap initialization :: nvme drives not empty after >12h
by Adrian Sevcenco 12 Mar '21

12 Mar '21

Hi! yesterday i bootstrapped (with cephadm) my first ceph installation and things looked somehow ok .. but today the osds are not yet ready and i have in dashboard this warnings: MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs PG_AVAILABILITY: Reduced data availability: 64 pgs inactive PG_DEGRADED: Degraded data redundancy: 2/14 objects degraded (14.286%), 66 pgs undersized TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3 and in logs: 3/12/21 12:18:19 PM [INF] OSD <1> is not empty yet. Waiting a bit more 3/12/21 12:18:19 PM [INF] OSD <0> is not empty yet. Waiting a bit more 3/12/21 12:18:19 PM [INF] Can't even stop one OSD. Cluster is probably busy. Retrying later.. 3/12/21 12:18:19 PM [ERR] cmd: osd ok-to-stop failed with: 31 PGs are already too degraded, would become too degraded or might become unavailable. (errno:-16) this is a single node, whole package ceph install with 2 local nvme drives as osds (to be used 2x replicated like a raid1 array) So, can anyone tell me what is going on? Thanks a lot!! Adrian

3 4

Recover data from Cephfs snapshot
by Jesper Lykkegaard Karlsen 12 Mar '21

12 Mar '21

Hi Ceph'ers, I love the possibility to make snapshots on Cephfs systems. Although there is one thing that puzzles me. Creating snapshot takes no time to do and deleting snapshots can bring PGs into snaptrim state for some hours. While recovering data from a snapshot will always invoke a full data transfer, where data are "physically" being copied back into place. This can make recovering from snapshots on Cephfs a rather heavy procedure. I have even tried "mv" command but that also starts transfer real data instead of just moving metadata pointers. Am I missing some "ceph snapshot recover" command, that can move metadata pointers and make recovery much lighter, or is this just that way it is? Best reagards, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka(a)mbg.au.dk Tlf: +45 50906203

1 0

mds rank failed. loaded with preallocated inodes that are inconsistent with inotable
by Ch Wan 12 Mar '21

12 Mar '21

Hi cephers. Recently we are upgrading ceph cluster from mimic to nautilus. We have 5 ranks and decrease max_mds from 5 down to 1 smoothly. When we set max_mds from 2 to 1, the cluster show that rank 1 is failed, those are mds logs: 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 handle_mds_map state change up:boot --> up:replay 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 replay_start 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 recovery set is 0 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 waiting for osdmap 460461 (which blacklists prior instance) 2021-03-12 16:21:27.018 7f366893d700 0 mds.1.cache creating system inode with ino:0x101 2021-03-12 16:21:27.019 7f366893d700 0 mds.1.cache creating system inode with ino:0x1 2021-03-12 16:21:27.404 7f366713a700 0 mds.1.cache creating system inode with ino:0x100 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client1: (2934972)loaded with preallocated inodes that are inconsistent with inotable 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client2: (2862164)loaded with preallocated inodes that are inconsistent with inotable 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client3: (2579839)loaded with preallocated inodes that are inconsistent with inotable 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client4: (2579815)loaded with preallocated inodes that are inconsistent with inotable ... thanks for any help!

1 0

Re: how to tell balancer to balance
by Boris Behrens 12 Mar '21

12 Mar '21

Hi Joe, I've tested to create a plan on my own, but I still get the same message (Error EALREADY: Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect). What I've also tried to reweight 0.8 the three most filled OSDs, which worked good. After all misplaced objects were resolved I reweighted them back to 1, and now I am where I started. The point with the wasted disks is something I can not follow. If there are two types of disks, ceph balances pretty good between these disks. -6 58.89636 - 59 TiB 41 TiB 40 TiB 2.4 GiB 104 GiB 18 TiB 69.20 0.98 - host s3db5 0 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 70 MiB 6.3 GiB 1.5 TiB 60.06 0.85 46 up osd.0 25 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 4.1 MiB 6.1 GiB 1.5 TiB 58.94 0.84 39 up osd.25 26 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 179 MiB 7.3 GiB 1.0 TiB 72.32 1.03 47 up osd.26 27 hdd 3.73630 1.00000 3.7 TiB 2.1 TiB 2.0 TiB 6.8 MiB 6.1 GiB 1.6 TiB 56.17 0.80 47 up osd.27 28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 935 MiB 28 GiB 3.9 TiB 73.68 1.05 197 up osd.28 29 hdd 14.65039 1.00000 15 TiB 9.9 TiB 9.8 TiB 515 MiB 24 GiB 4.7 TiB 67.83 0.96 188 up osd.29 30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 774 MiB 26 GiB 3.9 TiB 73.56 1.04 196 up osd.30 -7 58.89636 - 59 TiB 43 TiB 42 TiB 15 GiB 120 GiB 16 TiB 72.73 1.03 - host s3db6 32 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 22 MiB 7.8 GiB 1.0 TiB 72.18 1.02 60 up osd.32 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 381 MiB 8.1 GiB 670 GiB 82.50 1.17 57 up osd.33 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 444 MiB 8.5 GiB 604 GiB 84.21 1.20 60 up osd.34 35 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 296 MiB 8.6 GiB 513 GiB 86.59 1.23 53 up osd.35 36 hdd 14.65039 1.00000 15 TiB 9.9 TiB 9.8 TiB 541 MiB 24 GiB 4.8 TiB 67.55 0.96 209 up osd.36 37 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 12 GiB 36 GiB 4.5 TiB 69.22 0.98 191 up osd.37 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 4.0 TiB 72.62 1.03 209 up osd.38 [root@s3db1 ~]# ceph balancer optimize boristest Error EINVAL: Balancer enabled, disable to optimize manually [root@s3db1 ~]# ceph balancer off [root@s3db1 ~]# ceph balancer optimize bbtest Error EALREADY: Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect [root@s3db1 ~]# ceph balancer on Am Do., 11. März 2021 um 21:59 Uhr schrieb Joe Comeau <Joe.Comeau(a)hli.ubc.ca >: > I read your email > > maybe DO NOT use the balancer > pools should be of different disks > > then you could balance within the pools > > disks should all be the same size > the 4TB will fill up - the 8TB will only go to 50% (or 4 TB) - so in > effect wasting 4TB of the 8 TB disk > > our cluster & our pool > All our disks no matter what are 8 TB in size. > > > > > > >>> Boris Behrens <bb(a)kervyn.de> 3/11/2021 5:53 AM >>> > Hi, > I know this topic seems to be handled a lot (as far as I can see), but I > reached the end of my google_foo. > > * We have OSDs that are near full, but there are also OSDs that are only > loaded with 50%. > * We have 4,8,16 TB rotating disks in the cluster. > * The disks that get packed are 4TB disks and very empty disks are also 4TB > * The OSD nodes are all around the same total disk space (51 - 59) > * The balancer tells me that it can not find further optimization, or that > pg_num is decreasin. > > How can I debug further before the cluster goes into a bad state? > > [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | head -n 30 > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL > %USE VAR PGS STATUS TYPE NAME > MIN/MAX VAR: 0.75/1.23 STDDEV: 6.96 > TOTAL 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 > TiB 70.39 > -3 58.49872 - 58 TiB 39 TiB 36 TiB 8.2 GiB 85 GiB 19 > TiB 67.40 0.96 - host s3db2 > -4 58.49872 - 58 TiB 40 TiB 35 TiB 35 GiB 81 GiB 19 > TiB 67.91 0.96 - host s3db3 > -11 50.94173 - 51 TiB 35 TiB 35 TiB 3.5 GiB 94 GiB 16 > TiB 68.00 0.97 - host s3db10 > -10 51.28331 - 51 TiB 35 TiB 35 TiB 4.6 GiB 93 GiB 16 > TiB 69.16 0.98 - host s3db9 > -6 58.89636 - 59 TiB 41 TiB 40 TiB 2.4 GiB 102 GiB 18 > TiB 69.15 0.98 - host s3db5 > -12 50.99052 - 51 TiB 36 TiB 36 TiB 1.8 GiB 93 GiB 15 > TiB 69.99 0.99 - host s3db11 > -2 58.20561 - 58 TiB 41 TiB 37 TiB 9.6 GiB 96 GiB 17 > TiB 70.00 0.99 - host s3db1 > -1 673.44452 - 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 > TiB 70.39 1.00 - root default > -5 58.49872 - 58 TiB 42 TiB 35 TiB 7.0 GiB 94 GiB 17 > TiB 71.06 1.01 - host s3db4 > -8 58.89636 - 59 TiB 42 TiB 42 TiB 3.6 GiB 108 GiB 17 > TiB 71.91 1.02 - host s3db7 > -7 58.89636 - 59 TiB 43 TiB 42 TiB 15 GiB 120 GiB 16 > TiB 72.69 1.03 - host s3db6 > -37 58.55478 - 59 TiB 43 TiB 43 TiB 4.4 GiB 117 GiB 16 > TiB 73.18 1.04 - host s3db12 > -9 51.28331 - 51 TiB 38 TiB 38 TiB 4.9 GiB 103 GiB 13 > TiB 74.18 1.05 - host s3db8 > 15 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 2.1 GiB 0 B 1.7 > TiB 52.87 0.75 45 up osd.15 > 6 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 1.7 GiB 0 B 1.7 > TiB 52.90 0.75 46 up osd.6 > 12 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 570 MiB 0 B 1.7 > TiB 53.04 0.75 41 up osd.12 > 81 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 895 MiB 0 B 1.7 > TiB 54.26 0.77 51 up osd.81 > 27 hdd 3.73630 1.00000 3.7 TiB 2.1 TiB 2.0 TiB 6.8 MiB 5.8 GiB 1.6 > TiB 56.12 0.80 47 up osd.27 > 3 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.6 TiB 510 MiB 0 B 1.6 > TiB 57.04 0.81 51 up osd.3 > 5 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 431 MiB 0 B 1.5 > TiB 57.88 0.82 49 up osd.5 > 80 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 1.8 GiB 0 B 1.5 > TiB 58.31 0.83 51 up osd.80 > 25 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 4.1 MiB 6.1 GiB 1.5 > TiB 58.91 0.84 39 up osd.25 > 0 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 83 MiB 6.2 GiB 1.5 > TiB 60.03 0.85 46 up osd.0 > 79 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.4 TiB 1.8 GiB 0 B 1.4 > TiB 62.53 0.89 47 up osd.79 > 61 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 1.1 GiB 12 GiB 2.7 > TiB 62.80 0.89 101 up osd.61 > 67 hdd 7.27739 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 557 MiB 13 GiB 2.7 > TiB 63.29 0.90 96 up osd.67 > 72 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 107 MiB 11 GiB 2.7 > TiB 63.36 0.90 87 up osd.72 > > [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail > 51 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 724 MiB 14 GiB 1.7 > TiB 76.34 1.08 105 up osd.51 > 71 hdd 3.68750 1.00000 3.7 TiB 2.8 TiB 2.8 TiB 3.7 MiB 7.8 GiB 867 > GiB 77.04 1.09 47 up osd.71 > 82 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 839 GiB 628 MiB 0 B 839 > GiB 77.48 1.10 45 up osd.82 > 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 777 GiB 18 GiB 0 B 777 > GiB 79.14 1.12 59 up osd.14 > 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 752 GiB 826 MiB 0 B 752 > GiB 79.80 1.13 53 up osd.4 > 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 523 MiB 8.2 GiB 757 > GiB 79.95 1.14 53 up osd.75 > 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 237 MiB 9.2 GiB 668 > GiB 82.30 1.17 50 up osd.76 > 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 380 MiB 8.5 GiB 671 > GiB 82.46 1.17 57 up osd.33 > 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 464 MiB 8.4 GiB 605 > GiB 84.18 1.20 60 up osd.34 > 35 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 352 MiB 8.7 GiB 515 > GiB 86.55 1.23 53 up osd.35 > > [root@s3db1 ~]# ceph balancer status > { > "last_optimize_duration": "0:00:00.020142", > "plans": [], > "mode": "upmap", > "active": true, > "optimize_result": "Unable to find further optimization, or pool(s) > pg_num is decreasing, or distribution is already perfect", > "last_optimize_started": "Thu Mar 11 13:42:32 2021" > } > > [root@s3db1 ~]# ceph df > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 673 TiB 199 TiB 474 TiB 474 TiB 70.41 > TOTAL 673 TiB 199 TiB 474 TiB 474 TiB 70.41 > > POOLS: > POOL ID PGS STORED > OBJECTS USED %USED MAX AVAIL > rbd 0 64 0 B > 0 0 B 0 19 TiB > .rgw.root 1 64 98 KiB > 118 98 KiB 0 19 TiB > eu-central-1.rgw.control 2 64 0 B > 8 0 B 0 19 TiB > eu-central-1.rgw.data.root 3 64 1022 KiB > 3.02k 1022 KiB 0 19 TiB > eu-central-1.rgw.gc 4 64 84 MiB > 32 84 MiB 0 19 TiB > eu-central-1.rgw.log 5 64 220 MiB > 564 220 MiB 0 19 TiB > eu-central-1.rgw.users.uid 6 64 2.8 MiB > 6.89k 2.8 MiB 0 19 TiB > eu-central-1.rgw.users.keys 7 64 262 KiB > 6.70k 262 KiB 0 19 TiB > eu-central-1.rgw.meta 8 64 384 KiB > 1k 384 KiB 0 19 TiB > eu-central-1.rgw.users.email 9 64 40 B > 1 40 B 0 19 TiB > eu-central-1.rgw.buckets.index 10 64 10 GiB > 67.54k 10 GiB 0.02 19 TiB > eu-central-1.rgw.buckets.data 11 1024 151 TiB > 106.48M 151 TiB 72.67 19 TiB > eu-central-1.rgw.buckets.non-ec 12 64 268 MiB > 13.03k 268 MiB 0 19 TiB > eu-central-1.rgw.usage 13 64 501 MiB > 32 501 MiB 0 19 TiB > eu-msg-1.rgw.control 56 64 0 B > 8 0 B 0 19 TiB > eu-msg-1.rgw.data.root 57 64 71 KiB > 221 71 KiB 0 19 TiB > eu-msg-1.rgw.gc 58 64 60 KiB > 32 60 KiB 0 19 TiB > eu-msg-1.rgw.log 59 64 835 KiB > 242 835 KiB 0 19 TiB > eu-msg-1.rgw.users.uid 60 64 56 KiB > 107 56 KiB 0 19 TiB > eu-msg-1.rgw.usage 61 64 36 MiB > 25 36 MiB 0 19 TiB > eu-msg-1.rgw.users.keys 62 64 3.8 KiB > 97 3.8 KiB 0 19 TiB > eu-msg-1.rgw.meta 63 64 600 KiB > 1.58k 600 KiB 0 19 TiB > eu-msg-1.rgw.buckets.index 64 64 46 MiB > 112 46 MiB 0 19 TiB > eu-msg-1.rgw.users.email 65 64 0 B > 0 0 B 0 19 TiB > eu-msg-1.rgw.buckets.data 66 64 2.8 TiB > 1.14M 2.8 TiB 4.76 19 TiB > eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB > 353 2.2 MiB 0 19 TiB > default.rgw.control 69 32 0 B > 8 0 B 0 19 TiB > default.rgw.data.root 70 32 0 B > 0 0 B 0 19 TiB > default.rgw.gc 71 32 0 B > 0 0 B 0 19 TiB > default.rgw.log 72 32 0 B > 0 0 B 0 19 TiB > default.rgw.users.uid 73 32 0 B > 0 0 B 0 19 TiB > fra-1.rgw.control 74 32 0 B > 8 0 B 0 19 TiB > fra-1.rgw.meta 75 32 0 B > 0 0 B 0 19 TiB > fra-1.rgw.log 76 32 50 B > 28 50 B 0 19 TiB > whitespace-again-2021-03-10 77 64 111 MiB > 363.94k 111 MiB 0 19 TiB > whitespace-again-2021-03-10_2 78 32 18 KiB > 59 18 KiB 0 19 TiB > whitespace-again-2021-03-10_3 79 32 11 KiB > 36 11 KiB 0 19 TiB > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

ERROR: S3 error: 403 (SignatureDoesNotMatch)
by Szabo, Istvan (Agoda) 12 Mar '21

12 Mar '21

Hi, I'm struggling with my old cluster cnamed address. The s3 and curl commands are working properly with the not cnamed address, but with the cnamed one, I got this in the ciwetweb log: 2021-03-12 10:24:18.812329 7f6b0c527700 1 ====== starting new request req=0x7f6b0c520f90 ===== 2021-03-12 10:24:18.812387 7f6b0c527700 2 req 10:0.000058::HEAD /::initializing for trans_id = tx00000000000000000000a-00604adee2-8e4fc3-default 2021-03-12 10:24:18.812412 7f6b0c527700 10 rgw api priority: s3=5 s3website=4 2021-03-12 10:24:18.812417 7f6b0c527700 10 host=cnamedhostname 2021-03-12 10:24:18.812484 7f6b0c527700 10 handler=25RGWHandler_REST_Bucket_S3 2021-03-12 10:24:18.812490 7f6b0c527700 2 req 10:0.000163:s3:HEAD /::getting op 3 2021-03-12 10:24:18.812499 7f6b0c527700 10 op=25RGWStatBucket_ObjStore_S3 2021-03-12 10:24:18.812503 7f6b0c527700 2 req 10:0.000176:s3:HEAD /:stat_bucket:verifying requester 2021-03-12 10:24:18.812541 7f6b0c527700 2 req 10:0.000214:s3:HEAD /:stat_bucket:normalizing buckets and tenants 2021-03-12 10:24:18.812548 7f6b0c527700 10 s->object=<NULL> s->bucket= cnamedhostname 2021-03-12 10:24:18.812556 7f6b0c527700 2 req 10:0.000229:s3:HEAD /:stat_bucket:init permissions 2021-03-12 10:24:18.812594 7f6b0c527700 10 cache get: name=default.rgw.meta+root+ cnamedhostname : type miss (requested=0x16, cached=0x0) 2021-03-12 10:24:18.813525 7f6b0c527700 10 cache put: name=default.rgw.meta+root+ cnamedhostname info.flags=0x0 2021-03-12 10:24:18.813554 7f6b0c527700 10 moving default.rgw.meta+root+ cnamedhostname to cache LRU end 2021-03-12 10:24:18.813664 7f6b0c527700 10 read_permissions on cnamedhostname [] ret=-2002 2021-03-12 10:24:18.813833 7f6b0c527700 2 req 10:0.001506:s3:HEAD /:stat_bucket:op status=0 2021-03-12 10:24:18.813848 7f6b0c527700 2 req 10:0.001520:s3:HEAD /:stat_bucket:http status=404 2021-03-12 10:24:18.813855 7f6b0c527700 1 ====== req done req=0x7f6b0c520f90 op status=0 http_status=404 ====== 2021-03-12 10:24:18.813962 7f6b0c527700 1 civetweb: 0x557d45468000: 10.118.199.248 - - [12/Mar/2021:10:24:18 +0700] "HEAD / HTTP/1.1" 404 0 - curl/7.29.0 And I got this on the s3cmd verbose output: DEBUG: s3cmd version 2.1.0 DEBUG: ConfigParser: Reading file '.s3cfg-testuser-http' DEBUG: ConfigParser: access_key->29...17_chars...J DEBUG: ConfigParser: secret_key->fK...37_chars...R DEBUG: ConfigParser: host_base->cnamedhostname:80 DEBUG: ConfigParser: host_bucket->cnamedhostname:80/%(bucket) DEBUG: ConfigParser: use_https->False DEBUG: ConfigParser: signature_v2->True DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Unicodising 'ls' using UTF-8 DEBUG: Command: ls DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v2 DEBUG: SignHeaders: u'GET\n\n\n\nx-amz-date:Fri, 12 Mar 2021 03:31:39 +0000\n/' DEBUG: Processing request, please wait... DEBUG: get_hostname(None): cnamedhostname DEBUG: ConnMan.get(): creating new connection: http://cnamedhostname DEBUG: non-proxied HTTPConnection(cnamedhostname, None) DEBUG: format_uri(): / DEBUG: Sending request method_string='GET', uri=u'/', headers={'Authorization': u'AWS 293WEU2ADWGIUO4RN39J:Q7kh7kzWXWSqMvUqqWwLOY6QKUE=', 'x-amz-date': 'Fri, 12 Mar 2021 03:31:39 +0000'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://cnamedhostname#1) DEBUG: Response: {'data': '<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx00000000000000000000b-00604ae09b-8e4fbd-default</RequestId><HostId>8e4fbd-default-default</HostId></Error>', 'headers': {'accept-ranges': 'bytes', 'content-length': '198', 'content-type': 'application/xml', 'date': 'Fri, 12 Mar 2021 03:31:39 GMT', 'x-amz-request-id': 'tx00000000000000000000b-00604ae09b-8e4fbd-default'}, 'reason': 'Forbidden', 'status': 403} DEBUG: S3Error: 403 (Forbidden) DEBUG: HttpHeader: date: Fri, 12 Mar 2021 03:31:39 GMT DEBUG: HttpHeader: content-length: 198 DEBUG: HttpHeader: x-amz-request-id: tx00000000000000000000b-00604ae09b-8e4fbd-default DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: accept-ranges: bytes DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch' DEBUG: ErrorXML: RequestId: 'tx00000000000000000000b-00604ae09b-8e4fbd-default' DEBUG: ErrorXML: HostId: '8e4fbd-default-default' ERROR: S3 error: 403 (SignatureDoesNotMatch) Any idea? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 2

v14.2.17 Nautilus released
by David Galloway 11 Mar '21

11 Mar '21

Body: We're happy to announce the 17th backport release in the Nautilus series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-17-nautilus-released Notable Changes --------------- * $pid expansion in config paths like `admin_socket` will now properly expand to the daemon pid for commands like `ceph-mds` or `ceph-osd`. Previously only `ceph-fuse`/`rbd-nbd` expanded `$pid` with the actual daemon pid. * RADOS: PG removal has been optimized in this release. * RADOS: Memory allocations are tracked in finer detail in BlueStore and displayed as a part of the ``dump_mempools`` command. * cephfs: clients which acquire capabilities too quickly are throttled to prevent instability. See new config option ``mds_session_cap_acquisition_throttle`` to control this behavior. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.17.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 2e95b5d99e0dec516803c8a1b57fbd2c8f45fd63

1 0

Failure Domain = NVMe?
by Dave Hall 11 Mar '21

11 Mar '21

Hello, In some documentation I was reading last night about laying out OSDs, it was suggested that if more that one OSD uses the same NVMe drive, the failure-domain should probably be set to node. However, for a small cluster the inclination is to use EC-pools and failure-domain = OSD. I was wondering if there is a middle ground - could we define failure-domain = NVMe? I think the map would need to be defined manually in the same way that failure-domain = rack requires information about which nodes are in each rack. Example: My latest OSD nodes have 8 HDDs and 3 U.2 NVMe. I'd set up the WAL/DB for with HDDs per OSD (wasted space on the 3rd NVMe). Across all my OSD nodes I will have 8 HDDs and either 2 or 3 NVMe devices per node - 15 total NVMe devices. My preferred EC-pool profile is 8+2. It seems that this profile could be safely dispersed across 15 failure domains, resulting in protection against NVMe failure. Please let me know if this is worth pursuing. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

5 9

mon db growing. over 500Gb
by ricardo.re.azevedo＠gmail.com 11 Mar '21

11 Mar '21

Hi all, I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at: mon.a 89GB (the one that firt dropped out) mon.a 400GB mon.c 400GB I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again. Any idea what is going on? Or how can I shrik back down the db?

4 6

Can FS snapshots cause factor 3 performance loss?
by Frank Schilder 11 Mar '21

11 Mar '21

Hi all, we are observing a dramatic performance drop on our ceph file system and are wondering if this could be related to ceph fs snapshots. We are taking rotating snapshots in 2 directories and have 11 snapshots in each (ls below) as of today. We observe the performance drop with an rsync process that writes to ceph fs to another folder *without* snapshots. The performance reduction is a factor of 3 or even higher. Could this possibly be caused by the snapshots being present? Has anyone else seen something like this? The reason we consider snapshots is that not much else changed on the cluster except that we started taking rolling snapshots on the 23rd of February. In addition, the kernel symbols ceph_update_snap_trace, rebuild_snap_realms and build_snap_context show up really high in a perf report. The performance reduction seems to be present since at least 3 days. The ceph version is mimic 13.2.10. The kernel version of the rsync server is 3.10.0-1127.10.1.el7.x86_64. $ ls home/.snap 2021-02-23_183554+0100_weekly 2021-03-06_000611+0100_daily 2021-03-09_000611+0100_daily 2021-03-01_000911+0100_weekly 2021-03-07_000611+0100_daily 2021-03-10_000611+0100_daily 2021-03-04_000611+0100_daily 2021-03-08_000611+0100_daily 2021-03-11_000611+0100_daily 2021-03-05_000611+0100_daily 2021-03-08_000911+0100_weekly $ ls groups/.snap 2021-02-23_183554+0100_weekly 2021-03-06_000611+0100_daily 2021-03-09_000611+0100_daily 2021-03-01_000912+0100_weekly 2021-03-07_000611+0100_daily 2021-03-10_000612+0100_daily 2021-03-04_000611+0100_daily 2021-03-08_000611+0100_daily 2021-03-11_000612+0100_daily 2021-03-05_000611+0100_daily 2021-03-08_000911+0100_weekly Many thanks for any pointers and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Cephadm: Upgrade 15.2.5 -> 15.2.9 stops on non existing OSD
by Kai Stian Olstad 11 Mar '21

11 Mar '21

Before I started the upgrade the cluster was healthy but one OSD(osd.355) was down, can't remember if it was in or out. Upgrade was started with ceph orch upgrade start --image goharbor.example.com/library/ceph/ceph:v15.2.9 The upgrade started but when Ceph tried to upgrade osd.355 it paused with the following messages: 2021-03-11T09:15:35.638104+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Target is goharbor.example.com/library/ceph/ceph:v15.2.9 with id dfc48307963697ff48acd9dd6fda4a7a24017b9d8124f86c2 a542b0802fe77ba 2021-03-11T09:15:35.639882+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking mgr daemons... 2021-03-11T09:15:35.644170+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: All mgr daemons are up to date. 2021-03-11T09:15:35.644376+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking mon daemons... 2021-03-11T09:15:35.647669+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: All mon daemons are up to date. 2021-03-11T09:15:35.647866+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking crash daemons... 2021-03-11T09:15:35.652035+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Setting container_image for all crash... 2021-03-11T09:15:35.653683+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: All crash daemons are up to date. 2021-03-11T09:15:35.653896+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Checking osd daemons... 2021-03-11T09:15:36.273345+0000 mgr.pech-mon-2.cjeiyc [INF] It is presumed safe to stop ['osd.355'] 2021-03-11T09:15:36.273504+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: It is presumed safe to stop ['osd.355'] 2021-03-11T09:15:36.273887+0000 mgr.pech-mon-2.cjeiyc [INF] Upgrade: Redeploying osd.355 2021-03-11T09:15:36.276673+0000 mgr.pech-mon-2.cjeiyc [ERR] Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.355 on host pech-hd-009 failed. One of the first ting the upgrade did was to upgrade mon, so they are restarted and now the osd.355 no longer exist $ ceph osd info osd.355 Error EINVAL: osd.355 does not exist But if I run a resume ceph orch upgrade resume it still tries to upgrade osd.355, same message as above. I tried to stop and start the upgrade again with ceph orch upgrade stop ceph orch upgrade start --image goharbor.example.com/library/ceph/ceph:v15.2.9 it still tries to upgrade osd.355, with the same message as above. Looking at the source code it looks like it get daemons to upgrade from mgr cache, so I restarted both mgr but still it tries to upgrade osd.355. Does anyone know how I can get the upgrade to continue? -- Kai Stian Olstad

2 4

NVME pool creation time :: OSD services strange state
by Adrian Sevcenco 11 Mar '21

11 Mar '21

Hi! So, after i selected the tags to add 2 nvme ssds i declared a replicated n=2 pool .. and for the last 30 min the progress shown in notification is 0% and iotop shows around 100K/s for 2 (???) ceph-mon processes and that all ... and in my service list the osd services look somehow empty: https://prntscr.com/10iwwbh what did i miss? Thanks a lot! Adrian

1 2

Openstack rbd image Error deleting problem
by Norman.Kern 11 Mar '21

11 Mar '21

Hi Guys, I have used Ceph rbd for Openstack for sometime, I met a problem while destroying a VM. The Openstack tried to delete rbd image but failed. I have a test deleting a image by rbd command, it costs lots of time(image size 512G or more). Anyone met the same problem with me? Thanks, Norman

2 3

cephadm (curl master)/15.2.9:: how to add orchestration
by Adrian Sevcenco 11 Mar '21

11 Mar '21

Hi! After an initial bumpy bootstrapping (IMHO the defaults should be whatever is already defined in .ssh of the user and custom values setup with cli arguments) now i'm stuck adding any service/hosts/osds because apparently i lack orchestration .. the the documentation show a big "Page does not exist" see https://docs.ceph.com/en/latest/docs/octopus/mgr/orchestrator so, what is it and what options do i have? to set up it seems that is as easy as: ceph orch set backend I just started with ceph and i just want to start a ceph service (i cannot call it a cluster) on my desktop (with 2 dedicated osds) to get familiar also with usage. Thanks a lot! Adrian

4 4

how to tell balancer to balance
by Boris Behrens 11 Mar '21

11 Mar '21

Hi, I know this topic seems to be handled a lot (as far as I can see), but I reached the end of my google_foo. * We have OSDs that are near full, but there are also OSDs that are only loaded with 50%. * We have 4,8,16 TB rotating disks in the cluster. * The disks that get packed are 4TB disks and very empty disks are also 4TB * The OSD nodes are all around the same total disk space (51 - 59) * The balancer tells me that it can not find further optimization, or that pg_num is decreasin. How can I debug further before the cluster goes into a bad state? [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | head -n 30 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME MIN/MAX VAR: 0.75/1.23 STDDEV: 6.96 TOTAL 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 TiB 70.39 -3 58.49872 - 58 TiB 39 TiB 36 TiB 8.2 GiB 85 GiB 19 TiB 67.40 0.96 - host s3db2 -4 58.49872 - 58 TiB 40 TiB 35 TiB 35 GiB 81 GiB 19 TiB 67.91 0.96 - host s3db3 -11 50.94173 - 51 TiB 35 TiB 35 TiB 3.5 GiB 94 GiB 16 TiB 68.00 0.97 - host s3db10 -10 51.28331 - 51 TiB 35 TiB 35 TiB 4.6 GiB 93 GiB 16 TiB 69.16 0.98 - host s3db9 -6 58.89636 - 59 TiB 41 TiB 40 TiB 2.4 GiB 102 GiB 18 TiB 69.15 0.98 - host s3db5 -12 50.99052 - 51 TiB 36 TiB 36 TiB 1.8 GiB 93 GiB 15 TiB 69.99 0.99 - host s3db11 -2 58.20561 - 58 TiB 41 TiB 37 TiB 9.6 GiB 96 GiB 17 TiB 70.00 0.99 - host s3db1 -1 673.44452 - 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 TiB 70.39 1.00 - root default -5 58.49872 - 58 TiB 42 TiB 35 TiB 7.0 GiB 94 GiB 17 TiB 71.06 1.01 - host s3db4 -8 58.89636 - 59 TiB 42 TiB 42 TiB 3.6 GiB 108 GiB 17 TiB 71.91 1.02 - host s3db7 -7 58.89636 - 59 TiB 43 TiB 42 TiB 15 GiB 120 GiB 16 TiB 72.69 1.03 - host s3db6 -37 58.55478 - 59 TiB 43 TiB 43 TiB 4.4 GiB 117 GiB 16 TiB 73.18 1.04 - host s3db12 -9 51.28331 - 51 TiB 38 TiB 38 TiB 4.9 GiB 103 GiB 13 TiB 74.18 1.05 - host s3db8 15 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 2.1 GiB 0 B 1.7 TiB 52.87 0.75 45 up osd.15 6 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 1.7 GiB 0 B 1.7 TiB 52.90 0.75 46 up osd.6 12 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 570 MiB 0 B 1.7 TiB 53.04 0.75 41 up osd.12 81 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 895 MiB 0 B 1.7 TiB 54.26 0.77 51 up osd.81 27 hdd 3.73630 1.00000 3.7 TiB 2.1 TiB 2.0 TiB 6.8 MiB 5.8 GiB 1.6 TiB 56.12 0.80 47 up osd.27 3 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.6 TiB 510 MiB 0 B 1.6 TiB 57.04 0.81 51 up osd.3 5 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 431 MiB 0 B 1.5 TiB 57.88 0.82 49 up osd.5 80 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 1.8 GiB 0 B 1.5 TiB 58.31 0.83 51 up osd.80 25 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 4.1 MiB 6.1 GiB 1.5 TiB 58.91 0.84 39 up osd.25 0 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 83 MiB 6.2 GiB 1.5 TiB 60.03 0.85 46 up osd.0 79 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.4 TiB 1.8 GiB 0 B 1.4 TiB 62.53 0.89 47 up osd.79 61 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 1.1 GiB 12 GiB 2.7 TiB 62.80 0.89 101 up osd.61 67 hdd 7.27739 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 557 MiB 13 GiB 2.7 TiB 63.29 0.90 96 up osd.67 72 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 107 MiB 11 GiB 2.7 TiB 63.36 0.90 87 up osd.72 [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail 51 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 724 MiB 14 GiB 1.7 TiB 76.34 1.08 105 up osd.51 71 hdd 3.68750 1.00000 3.7 TiB 2.8 TiB 2.8 TiB 3.7 MiB 7.8 GiB 867 GiB 77.04 1.09 47 up osd.71 82 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 839 GiB 628 MiB 0 B 839 GiB 77.48 1.10 45 up osd.82 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 777 GiB 18 GiB 0 B 777 GiB 79.14 1.12 59 up osd.14 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 752 GiB 826 MiB 0 B 752 GiB 79.80 1.13 53 up osd.4 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 523 MiB 8.2 GiB 757 GiB 79.95 1.14 53 up osd.75 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 237 MiB 9.2 GiB 668 GiB 82.30 1.17 50 up osd.76 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 380 MiB 8.5 GiB 671 GiB 82.46 1.17 57 up osd.33 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 464 MiB 8.4 GiB 605 GiB 84.18 1.20 60 up osd.34 35 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 352 MiB 8.7 GiB 515 GiB 86.55 1.23 53 up osd.35 [root@s3db1 ~]# ceph balancer status { "last_optimize_duration": "0:00:00.020142", "plans": [], "mode": "upmap", "active": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "last_optimize_started": "Thu Mar 11 13:42:32 2021" } [root@s3db1 ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 673 TiB 199 TiB 474 TiB 474 TiB 70.41 TOTAL 673 TiB 199 TiB 474 TiB 474 TiB 70.41 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL rbd 0 64 0 B 0 0 B 0 19 TiB .rgw.root 1 64 98 KiB 118 98 KiB 0 19 TiB eu-central-1.rgw.control 2 64 0 B 8 0 B 0 19 TiB eu-central-1.rgw.data.root 3 64 1022 KiB 3.02k 1022 KiB 0 19 TiB eu-central-1.rgw.gc 4 64 84 MiB 32 84 MiB 0 19 TiB eu-central-1.rgw.log 5 64 220 MiB 564 220 MiB 0 19 TiB eu-central-1.rgw.users.uid 6 64 2.8 MiB 6.89k 2.8 MiB 0 19 TiB eu-central-1.rgw.users.keys 7 64 262 KiB 6.70k 262 KiB 0 19 TiB eu-central-1.rgw.meta 8 64 384 KiB 1k 384 KiB 0 19 TiB eu-central-1.rgw.users.email 9 64 40 B 1 40 B 0 19 TiB eu-central-1.rgw.buckets.index 10 64 10 GiB 67.54k 10 GiB 0.02 19 TiB eu-central-1.rgw.buckets.data 11 1024 151 TiB 106.48M 151 TiB 72.67 19 TiB eu-central-1.rgw.buckets.non-ec 12 64 268 MiB 13.03k 268 MiB 0 19 TiB eu-central-1.rgw.usage 13 64 501 MiB 32 501 MiB 0 19 TiB eu-msg-1.rgw.control 56 64 0 B 8 0 B 0 19 TiB eu-msg-1.rgw.data.root 57 64 71 KiB 221 71 KiB 0 19 TiB eu-msg-1.rgw.gc 58 64 60 KiB 32 60 KiB 0 19 TiB eu-msg-1.rgw.log 59 64 835 KiB 242 835 KiB 0 19 TiB eu-msg-1.rgw.users.uid 60 64 56 KiB 107 56 KiB 0 19 TiB eu-msg-1.rgw.usage 61 64 36 MiB 25 36 MiB 0 19 TiB eu-msg-1.rgw.users.keys 62 64 3.8 KiB 97 3.8 KiB 0 19 TiB eu-msg-1.rgw.meta 63 64 600 KiB 1.58k 600 KiB 0 19 TiB eu-msg-1.rgw.buckets.index 64 64 46 MiB 112 46 MiB 0 19 TiB eu-msg-1.rgw.users.email 65 64 0 B 0 0 B 0 19 TiB eu-msg-1.rgw.buckets.data 66 64 2.8 TiB 1.14M 2.8 TiB 4.76 19 TiB eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB 353 2.2 MiB 0 19 TiB default.rgw.control 69 32 0 B 8 0 B 0 19 TiB default.rgw.data.root 70 32 0 B 0 0 B 0 19 TiB default.rgw.gc 71 32 0 B 0 0 B 0 19 TiB default.rgw.log 72 32 0 B 0 0 B 0 19 TiB default.rgw.users.uid 73 32 0 B 0 0 B 0 19 TiB fra-1.rgw.control 74 32 0 B 8 0 B 0 19 TiB fra-1.rgw.meta 75 32 0 B 0 0 B 0 19 TiB fra-1.rgw.log 76 32 50 B 28 50 B 0 19 TiB whitespace-again-2021-03-10 77 64 111 MiB 363.94k 111 MiB 0 19 TiB whitespace-again-2021-03-10_2 78 32 18 KiB 59 18 KiB 0 19 TiB whitespace-again-2021-03-10_3 79 32 11 KiB 36 11 KiB 0 19 TiB -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: how smart is ceph recovery?
by Marc 11 Mar '21

11 Mar '21

> > > > 2. If a down host comes up again and it's osd are started. Is data > still being copied, or does ceph see that checksums(?) > > PG or RADOS object epoch I think. So if data hasn’t changed, the > recovery completes without having anything to do. > > > are the same and just sets a pointer(?) > back to the old location? Yes I mean the osdmap having the old osd on the node that was down. Hmmm, I currently have pg's in 'active+remapped+backfill_wait' of a pool rbd.backup. Of which I know nothing in it has changed. And those osd's listed[1] are already up. Especially when you are getting at the 'end' of recovery where osd_max_backfills=X has less effect and recovery takes longer. It would be nice to have the "no work needed pg's" be processed quickly. [1] [18,25,41]p18 [18,41,17] [5,0,25]p5 [5,0,4

1 0

Unpurgeable rbd image from trash
by Enrico Bocchi 11 Mar '21

11 Mar '21

Hello everyone, We have an unpurgeable image living in the trash of one of our clusters: # rbd --pool volumes trash ls 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b If we try to purge the whole trash it says the image is being restored but we have never tried to do that: # rbd --pool volumes trash purge Removing images: 0% complete...failed. 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: error: image is pending restoration. When trying to delete manually, it says there are some watchers, but this is actually not the case: # rbd --pool volumes trash remove 5afa5e5a07b8bc rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 -1 librbd::api::Trash: remove: error: image is pending restoration. This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. Removing image: 0% complete...failed. # rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc # We have tried to stat the first 10 rbd_data objects and they were all deleted. We know we can manually delete the omapkey from rbd_trash but we though it would be better to understand how an image might get in this state. Has anyone seen this before? Many thanks! Cheers, Enrico -- Enrico Bocchi CERN European Laboratory for Particle Physics IT - Storage Group - General Storage Services Mailbox: G20500 - Office: 31-2-010 1211 Genève 23 Switzerland

2 4

A practical approach to efficiently store 100 billions small objects in Ceph
by Loïc Dachary 11 Mar '21

11 Mar '21

Bonjour, In the past weeks a few mailing list threads[0][1][2] explored the problem of storing billions of small objects in Ceph. There was great feedback (I learned at lot) and it turns out the solution is a rather simple aggregation of the ideas that were suggested during these discussions. It is described in detail here: https://wiki.softwareheritage.org/wiki/A_practical_approach_to_efficiently_… The next step will be to write and run benchmarks[3]. Although I'm convinced it is a good solution, I've been wrong before and confirmation is required :-) Many thanks to all the participants in the discussions on behalf of the Software Heritage project[4]. Cheers [0] Storing 20 billions of immutable objects in Ceph, 75% <16KB https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/JSG2TXKNXPX… [1] Small RGW objects and RADOS 64KB minimun size https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJF… [2] Using RBD to pack billions of small files https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISX… [3] https://forge.softwareheritage.org/T3054 [4] https://www.softwareheritage.org/ -- Loïc Dachary, Artisan Logiciel Libre

4 8

RadosGW unable to start resharding
by Ansgar Jazdzewski 11 Mar '21

11 Mar '21

Hi Folks, We are running ceph 14.2.16 and I like to reshard a bucket because I have a large object warning! so I did: radosgw-admin bucket reshard --tenant="..." --bucket="..." --uid="..." --num-shards=512 but I got receive an error: ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time `radosgw-admin reshard list` is empty so I assume I have to delete some leftovers from the old resharding!? did someone has had this before? thanks for your input, Ansgar

2 3

Bluestore OSD Layout - WAL, DB, Journal
by Dave Hall 11 Mar '21

11 Mar '21

Hello, I'm in the process of doubling the number of OSD nodes in my Nautilus cluster - from 3 to 6. Based on answers receive from earlier posts to this list, the new nodes have more NVMe that the old nodes. More to the point, on the original nodes the amount of NVMe allocated to each OSD was about 120GB, so the RocksDB was limited to 30GB. However, for my workload 300GB is probably recommended. As I prepare to lay out the NVMe on these new nodes, I'm still trying to understand how to size the DB and WAL for my OSDs and whether Journal is even needed. According to https://docs.ceph.com/en/nautilus/ceph-volume/lvm/prepare/ <https://docs.ceph.com/en/nautilus/ceph-volume/lvm/prepare/> > Bluestore supports the following configurations: > > * A block device, a block.wal, and a block.db device > * A block device and a block.wal device > * A block device and a block.db device > * A single block device > First question: On my first nodes I managed to get a DB, but no WAL. My current perception is that WAL and DB occupy separate physical/logical partitions. By specifying a WAL size and a DB size, ceph-volume will create the corresponding logical volumes on the NVMe. Is this correct? It is also possible to lay these out as basic logical partitions? Second question: How do I decide whether I need WAL, DB, or both? Third question: Once I answer the above WAL/DB question, what are the guidelines for sizing them? Thanks. -Dave -- Dave Hall Binghamton University

2 1

Some confusion around PG, OSD and balancing issue
by Darrin Hodges 11 Mar '21

11 Mar '21

HI all, Just looking for clarification around the relationship between PGs,OSDs and balancing on a ceph (octopus) cluster. We have pg autobalance on and balancing is set to upmap. There are 2 pools, one is the default metric pool with 1 pg, the other is the pool we are using for everything, it has 512 PG. There are 60 OSD's split across 4 hosts. The USD usage ranges between 39% and 68%: * current cluster score 0.048908 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.3 GiB 1.5 TiB 47.08 0.86 19 up 1 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 3.2 MiB 2.3 GiB 1.4 TiB 49.46 0.90 20 up 2 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.8 MiB 2.4 GiB 1.3 TiB 51.81 0.95 21 up 3 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 589 KiB 2.4 GiB 1.3 TiB 51.73 0.95 21 up 4 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.2 MiB 2.7 GiB 1.1 TiB 59.05 1.08 24 up 5 hdd 9.00000 1.00000 9.1 TiB 5.2 TiB 5.2 TiB 6.9 MiB 8.2 GiB 3.9 TiB 56.75 1.04 77 up 6 hdd 2.74550 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.9 MiB 2.8 GiB 1.3 TiB 54.21 0.99 22 up 7 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.7 GiB 1.4 TiB 49.32 0.90 20 up 8 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.3 GiB 1.4 TiB 49.36 0.90 20 up 9 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 3.3 MiB 3.0 GiB 1.1 TiB 59.21 1.08 24 up 10 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.2 MiB 3.4 GiB 941 GiB 66.53 1.22 27 up 11 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.9 MiB 2.0 GiB 1.7 TiB 39.69 0.73 16 up 12 hdd 2.74550 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 1.2 MiB 2.1 GiB 1.5 TiB 44.69 0.82 18 up 13 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 2.4 MiB 1.9 GiB 1.7 TiB 39.59 0.72 16 up 14 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.1 MiB 3.1 GiB 945 GiB 66.37 1.21 27 up 15 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 1.7 MiB 2.6 GiB 1.2 TiB 56.68 1.04 23 up 16 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 3.3 MiB 2.3 GiB 1.5 TiB 46.76 0.86 19 up 17 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 2.8 MiB 3.1 GiB 953 GiB 66.12 1.21 26 up 18 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 GiB 1010 GiB 64.08 1.17 26 up 19 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 3.0 MiB 2.9 GiB 1016 GiB 63.88 1.17 26 up 20 hdd 9.00000 1.00000 9.1 TiB 4.7 TiB 4.7 TiB 28 MiB 7.5 GiB 4.4 TiB 51.47 0.94 70 up 21 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 2.8 MiB 2.2 GiB 1.5 TiB 44.43 0.81 18 up 22 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 653 KiB 2.4 GiB 1.3 TiB 51.89 0.95 21 up 23 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 GiB 1.1 TiB 59.26 1.08 24 up 24 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 3.1 MiB 3.0 GiB 1.2 TiB 56.67 1.04 23 up 25 hdd 2.74599 0.90002 2.7 TiB 1.7 TiB 1.7 TiB 5.3 MiB 4.0 GiB 1.1 TiB 61.47 1.12 25 up 26 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 1.5 MiB 2.4 GiB 1.3 TiB 51.82 0.95 21 up 27 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.1 MiB 2.2 GiB 1.5 TiB 46.93 0.86 19 up 28 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.6 MiB 2.2 GiB 1.5 TiB 46.88 0.86 19 up 29 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 557 KiB 2.6 GiB 1.4 TiB 49.37 0.90 20 up 45 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 GiB 1.1 TiB 59.18 1.08 24 up 46 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 28 MiB 2.5 GiB 1.3 TiB 51.76 0.95 22 up 47 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 977 KiB 3.1 GiB 1.1 TiB 59.07 1.08 24 up 48 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 625 KiB 2.6 GiB 1.5 TiB 46.86 0.86 19 up 49 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.9 MiB 3.2 GiB 1.1 TiB 61.68 1.13 25 up 50 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.0 MiB 2.7 GiB 1.1 TiB 61.53 1.13 25 up 51 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.1 TiB 2.5 MiB 2.3 GiB 1.6 TiB 41.88 0.77 17 up 52 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 1.8 MiB 3.8 GiB 942 GiB 66.49 1.22 27 up 53 hdd 9.00000 1.00000 9.1 TiB 4.3 TiB 4.3 TiB 6.2 MiB 6.8 GiB 4.8 TiB 47.46 0.87 64 up 54 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 GiB 1008 GiB 64.13 1.17 26 up 55 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 GiB 1014 GiB 63.95 1.17 26 up 56 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.8 GiB 1013 GiB 63.99 1.17 26 up 57 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 2.9 MiB 2.6 GiB 1.2 TiB 56.67 1.04 23 up 58 hdd 2.74599 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.2 MiB 2.5 GiB 1.3 TiB 54.41 1.00 22 up 59 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 GiB 1015 GiB 63.92 1.17 26 up 30 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 1.6 MiB 3.1 GiB 878 GiB 68.79 1.26 28 up 31 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.2 GiB 1.5 TiB 47.03 0.86 19 up 32 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 3.2 MiB 3.3 GiB 944 GiB 66.45 1.22 27 up 33 hdd 2.74599 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.1 MiB 1.9 GiB 1.7 TiB 39.88 0.73 16 up 34 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 28 MiB 3.0 GiB 874 GiB 68.93 1.26 29 up 35 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.3 MiB 2.2 GiB 1.5 TiB 47.01 0.86 19 up 36 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 1.6 MiB 3.0 GiB 947 GiB 66.31 1.21 27 up 37 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 1.8 MiB 2.9 GiB 1.1 TiB 58.96 1.08 24 up 38 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 2.8 MiB 3.4 GiB 877 GiB 68.82 1.26 28 up 39 hdd 2.74599 0.85004 2.7 TiB 1.6 TiB 1.6 TiB 2.4 MiB 3.5 GiB 1.1 TiB 59.20 1.08 24 up 40 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.5 MiB 2.3 GiB 1.4 TiB 49.35 0.90 20 up 41 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.3 MiB 2.4 GiB 1.3 TiB 51.73 0.95 21 up 42 hdd 9.00000 1.00000 9.1 TiB 4.5 TiB 4.5 TiB 2.1 MiB 7.2 GiB 4.6 TiB 49.76 0.91 67 up 43 hdd 2.74599 0.90002 2.7 TiB 1.6 TiB 1.6 TiB 2.9 MiB 3.2 GiB 1.1 TiB 59.17 1.08 24 up 44 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 4.0 MiB 2.2 GiB 1.5 TiB 44.34 0.81 18 up 54 of the OSD are 3TiB and there are four 8TiB - can this cause imbalance? Or do I need to increase the PG from 512? or is this as good as it gets? many thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

1 0

how smart is ceph recovery?
by Marc 10 Mar '21

10 Mar '21

1. is backfilling/remapping so smart that it will do what ever it can? Or are there situations like a pg a. is scheduled to be moved, but can not be moved because of min_size. Now another pg b. cannot be moved because pg a. allocated osd space and the backfill ratio will be met. Yet if the order was reversed if the pg b. would be moved and pg a. would not be moved because it is stuck anyway and the backfill ratio will be met. 2. If a down host comes up again and it's osd are started. Is data still being copied, or does ceph see that checksums(?) are the same and just sets a pointer(?) back to the old location? > -----Original Message----- > Sent: 09 March 2021 23:59 > To: ceph-users(a)ceph.io > Subject: [ceph-users] node down pg with backfill_wait waiting for > incomplete? > > > I have a node down and pg's are remapping/backfilling. I have also a lot > of pg's in backfill_wait. > > I was wondering if there is a specific order that this is being > executed. Eg I have a large'garbage' pool ec21 that is stuck. I could > resolve that by changing the min size. However I rather have this not > remapped, and wait for this node to be back on line. > > Question is: is other remapping/backfilling waiting for this stuck to be > fixed, or is backfilling/remapping so smart that it will do what ever it > can? > > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. > F1 Outsourcing Development Sp. z o.o. > Poland > > t: +48 (0)12 4466 845 > f: +48 (0)12 4466 843 > e: marc(a)f1-outsourcing.eu > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

How to speed up removing big rbd pools
by huxiaoyu＠horebdata.cn 10 Mar '21

10 Mar '21

Dear Cepher， For some reasons, i had a cluster with several 20 TB pools and 100TB ones, which were previously linked with iSCSI for virtual machines. When deleting those big rbd images, it turns out to be extremely slow, taking hours if not days. The Ceph cluster is running on Luminous 12.2.13 with bluestore How to speed up removing big rbd pools? thanks, samuel huxiaoyu(a)horebdata.cn

1 0

PG inactive when host is down despite CRUSH failure domain being host
by Janek Bevendorff 10 Mar '21

10 Mar '21

Hi, I am having a weird phenomenon, which I am having trouble to debug. We have 16 OSDs per host, so when I reboot one node, 16 OSDs will be missing for a short time. Since our minimum CRUSH failure domain is host, this should not cause any problems. Unfortunately, I always have handful (1-5) PGs that become inactive nonetheless and are stuck in the state undersized+degraded+peered until the host and its OSDs are back up. The other 2000+ PGs that are also on these OSDs do not have this problem. In total, we have between 110 and 150 PGs per OSD with a configured maximum of 250, which should give us enough headspace. The affected pools always seem to be RBD pools or at least I haven't seen it on our much larger RGW pool yet. The pool's CRUSH rule looks like this: rule rbd-data { id 8 type replicated min_size 2 max_size 10 step take default step chooseleaf firstn 0 type host step emit } ceph pg dump_stuck inactive gives me this: PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 115.3 undersized+degraded+peered [194,267] 194 [194,267] 194 115.13 undersized+degraded+peered [151,1122] 151 [151,1122] 151 116.12 undersized+degraded+peered [288,726] 288 [288,726] 288 and when I query one of the inactive PGs, I see (among other things): "up": [ 288, 726 ], "acting": [ 288, 726 ], "acting_recovery_backfill": [ "288", "726" ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2021-03-10T16:23:09.301174+0100", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "pull_from_peer": [], "pushing": [] } } }, { "name": "Started", "enter_time": "2021-03-10T16:23:08.297622+0100" } ], So you can see that two out of three OSDs on other hosts are indeed up and active and the . I also see the ceph-osd daemons running on those hosts, so the data is definitely there and the PG should be available. Do you have any idea why these PGs may be becoming inactive nonetheless? I am suspecting some kind of concurrency limit, but I wouldn't know which one that could be. Thanks Janek

2 2

OSD crashes create_aligned_in_mempool in 15.2.9 and 14.2.16
by Andrej Filipcic 10 Mar '21

10 Mar '21

Hi, under heavy load our cluster is experiencing frequent OSD crashes. Is this a known bug or should I report it? Any workarounds? It looks to be highly correlated with memory tuning. it happens with both nautilus 14.2.16 and octopus 15.2.9. I have forced the bitmap bluefs and bluestore allocator. the cluster is ~60 nodes with 256GB ram and 25Gb NICs, ~1600 OSDs on 100g network. Typically it is happening when the traffic is above 100GB/s. Best regards, Andrej -14> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient: tick -13> 2021-03-09T14:10:30.105+0100 7fc128e05700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-03-09T14:10:00.107344+0100) -12> 2021-03-09T14:10:30.210+0100 7fc119412700 5 osd.209 9539 heartbeat osd_stat(store_statfs(0xe68762a0000/0x40000000/0xe8d7fc00000, data 0x24b195d923/0x24c97c0000, compress 0x0/0x0/0x0, omap 0xf5dca, meta 0x3ff0a236), peers [6,8,11,21,22,23,24,25,26,27,28,29,34,35,37,38,65,86,87,90,120,128,129,135,136,140,150,153,154,160,184,188,192,193,203,208,210,217,229,233,242,248,254,25 6,275,277,282,290,311,313,324,326,331,339,348,369,409,411,413,466,477,532,535,538,539,542,544,546,548,552,554,556,558,561,576,580,600,601,604,612,614,624,625,631,657,689,695,704,717,738,739,740,766,790,810,812,833,839,890,891,895,903,909,916,926,927,946,960,965,991,1050,1055,1062,1064,1067,1069,1072,1073,1075,1077,1078,1079,1095,1100,1117,1125,1127,1141,1148,1149,1153,1155,1195,12 01,1202,1215,1229,1238,1253,1260,1283,1290,1298,1303,1329,1330,1349,1350,1388,1389,1422,1423,1430,1431,1434,1448,1455,1478,1479,1485,1488,1494,1497,1506,1516,1561,1564,1573,1574,1580] op hist [0,0,0,1,3,4,15,24,43,64,102,117]) -11> 2021-03-09T14:10:30.468+0100 7fc137cc0700 10 monclient: handle_auth_request added challenge on 0x55c08becf800 -10> 2021-03-09T14:10:30.543+0100 7fc1374bf700 10 monclient: handle_auth_request added challenge on 0x55c08becf400 -9> 2021-03-09T14:10:30.712+0100 7fc1384c1700 10 monclient: handle_auth_request added challenge on 0x55c08becec00 -8> 2021-03-09T14:10:31.029+0100 7fc137cc0700 10 monclient: handle_auth_request added challenge on 0x55c08becfc00 -7> 2021-03-09T14:10:31.033+0100 7fc12ca33700 5 prioritycache tune_memory target: 7264711979 mapped: 1564606464 unmapped: 47874048 heap: 1612480512 old mem: 5369698813 new mem: 5369698813 -6> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient: tick -5> 2021-03-09T14:10:31.105+0100 7fc128e05700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-03-09T14:10:01.107451+0100) -4> 2021-03-09T14:10:31.574+0100 7fc1374bf700 10 monclient: handle_auth_request added challenge on 0x55c0b69a8000 -3> 2021-03-09T14:10:32.036+0100 7fc12ca33700 5 prioritycache tune_memory target: 7264711979 mapped: 1708449792 unmapped: 46637056 heap: 1755086848 old mem: 5369698813 new mem: 5369698813 -2> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient: tick -1> 2021-03-09T14:10:32.106+0100 7fc128e05700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-03-09T14:10:02.107524+0100) 0> 2021-03-09T14:10:32.661+0100 7fc1384c1700 -1 *** Caught signal (Aborted) ** in thread 7fc1384c1700 thread_name:msgr-worker-0 ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable) 1: (()+0x12b20) [0x7fc13cc20b20] 2: (gsignal()+0x10f) [0x7fc13b8847ff] 3: (abort()+0x127) [0x7fc13b86ec35] 4: (()+0x9009b) [0x7fc13c23a09b] 5: (()+0x9653c) [0x7fc13c24053c] 6: (()+0x96597) [0x7fc13c240597] 7: (()+0x967f8) [0x7fc13c2407f8] 8: (ceph::buffer::v15_2_0::create_aligned_in_mempool(unsigned int, unsigned int, int)+0x229) [0x55c0669dce49] 9: (ceph::buffer::v15_2_0::create_aligned(unsigned int, unsigned int)+0x26) [0x55c0669dcf46] 10: (ceph::buffer::v15_2_0::create_small_page_aligned(unsigned int)+0x55) [0x55c0669dd8e5] 11: (ProtocolV1::read_message_data_prepare()+0x368) [0x55c066b78ac8] 12: (ProtocolV1::read_message_middle()+0x130) [0x55c066b78c90] 13: (ProtocolV1::handle_message_front(char*, int)+0x2f4) [0x55c066b79624] 14: (()+0xf72bed) [0x55c066b72bed] 15: (AsyncConnection::process()+0x8a9) [0x55c066b6fa39] 16: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x55c0669c41b7] 17: (()+0xdc979c) [0x55c0669c979c] 18: (()+0xc2ba3) [0x7fc13c26cba3] 19: (()+0x814a) [0x7fc13cc1614a] 20: (clone()+0x43) [0x7fc13b949f23] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_rwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7fc119412700 / osd_srv_heartbt 7fc119c13700 / tp_osd_tp 7fc11a414700 / tp_osd_tp 7fc11ac15700 / tp_osd_tp 7fc11b416700 / tp_osd_tp 7fc11bc17700 / tp_osd_tp 7fc124c29700 / ms_dispatch 7fc125c2b700 / rocksdb:dump_st 7fc12822a700 / bstore_kv_sync 7fc128e05700 / safe_timer 7fc129e07700 / ms_dispatch 7fc12ca33700 / bstore_mempool 7fc133446700 / safe_timer 7fc1374bf700 / msgr-worker-2 7fc137cc0700 / msgr-worker-1 7fc1384c1700 / msgr-worker-0 max_recent 10000 max_new 1000 -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

3 4

Re: Rados gateway basic pools missing
by St-Germain, Sylvain (SSC/SPC) 10 Mar '21

10 Mar '21

Ok I fix it works -----Message d'origine----- De : St-Germain, Sylvain (SSC/SPC) <sylvain.st-germain(a)canada.ca> Envoyé : 9 mars 2021 17:41 À : St-Germain, Sylvain (SSC/SPC) <sylvain.st-germain(a)canada.ca>; ceph-users(a)ceph.io Objet : RE: Rados gateway basic pools missing Ok in the interface when I create a bucket the index in created automatically 1 device_health_metrics 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.log 6 default.rgw.control 7 default.rgw.meta 8 default.rgw.buckets.index * I think I just could not make an insertion using s3cmd List command - connection problem # s3cmd la !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please try reproducing the error using the latest s3cmd code from the git master branch found at: https://github.com/s3tools/s3cmd and have a look at the known issues list: https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutio… If the error persists, please report the following lines (removing any private info as necessary) to: s3tools-bugs(a)lists.sourceforge.net !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Invoked as: /usr/bin/s3cmd la Problem: <class 'ConnectionRefusedError: [Errno 111] Connection refused S3cmd: 2.0.2 python: 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0] environment LANG=en_CA.UTF-8 Traceback (most recent call last): File "/usr/bin/s3cmd", line 3092, in <module> rc = main() File "/usr/bin/s3cmd", line 3001, in main rc = cmd_func(args) File "/usr/bin/s3cmd", line 164, in cmd_all_buckets_list_all_content response = s3.list_all_buckets() File "/usr/lib/python3/dist-packages/S3/S3.py", line 302, in list_all_buckets response = self.send_request(request) File "/usr/lib/python3/dist-packages/S3/S3.py", line 1258, in send_request conn = ConnMan.get(self.get_hostname(resource['bucket'])) File "/usr/lib/python3/dist-packages/S3/ConnMan.py", line 253, in get conn.c.connect() File "/usr/lib/python3.8/http/client.py", line 921, in connect self.sock = self._create_connection( File "/usr/lib/python3.8/socket.py", line 808, in create_connection raise err File "/usr/lib/python3.8/socket.py", line 796, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please try reproducing the error using the latest s3cmd code from the git master branch found at: https://github.com/s3tools/s3cmd and have a look at the known issues list: https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutio… If the error persists, please report the above lines (removing any private info as necessary) to: s3tools-bugs(a)lists.sourceforge.net !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -----Message d'origine----- De : St-Germain, Sylvain (SSC/SPC) <sylvain.st-germain(a)canada.ca> Envoyé : 9 mars 2021 17:19 À : ceph-users(a)ceph.io Objet : [ceph-users] Rados gateway basic pools missing Hi everyone, I just rebuild a (test) cluster using : OS : Ubuntu 20.04.2 LTS CEPH : ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable) 3 nodes : monitor/storage 1. The cluster looks good : # ceph -s cluster: id: 9a89aa5a-1702-4f87-a99c-f94c9f2cdabd health: HEALTH_OK services: mon: 3 daemons, quorum dao-wkr-04,dao-wkr-05,dao-wkr-06 (age 7m) mgr: dao-wkr-05(active, since 8m), standbys: dao-wkr-04, dao-wkr-06 mds: cephfs:1 {0=dao-wkr-04=up:active} 2 up:standby osd: 9 osds: 9 up (since 7m), 9 in (since 4h) rgw: 3 daemons active (dao-wkr-04.rgw0, dao-wkr-05.rgw0, dao-wkr-06.rgw0) task status: data: pools: 7 pools, 121 pgs objects: 234 objects, 16 KiB usage: 9.0 GiB used, 2.0 TiB / 2.0 TiB avail pgs: 121 active+clean 2. except that the main pools for the radosgw are not there # sudo ceph osd lspools 1 device_health_metrics 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.log 6 default.rgw.control 7 default.rgw.meta Missing : default.rgw.buckets.index & default.rgw.buckets.data What do you think ? Thx ! Sylvain _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

buckets with negative num_objects
by Boris Behrens 10 Mar '21

10 Mar '21

Hi, I am in the process of resharding large buckets and to find them I ran radosgw-admin bucket limit check | grep '"fill_status": "OVER' -B5 and I see that there are two buckets with negative num_objects "bucket": "ncprod", "tenant": "", "num_objects": -482, "num_shards": 0, "objects_per_shard": -482, "fill_status": "OVER 100.000000%" -- "bucket": "fileshare-s3", "tenant": "", "num_objects": -137, "num_shards": 0, "objects_per_shard": -137, "fill_status": "OVER 100.000000%" Is this an error? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

ceph pool with a whitespace as name
by Boris Behrens 10 Mar '21

10 Mar '21

Good morning ceph people, I have a pool that got a whitespace as name. And I want to know what creates the pool. I already renamed it, but something recreates the pool. Is there a way to find out what created the pool and what the content ist? When I checked it's content I get [root@s3db1 ~]# rados -p " " ls .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86469149.1766:_Feueqyht8adm09IVoqhR7Sh:1 .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86450896.1831:_rU77PL7UJkTlOovtCWkvnfP:1 ... [root@s3db1 ~]# rados -p " " listxattr .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 ceph.objclass.version user.rgw.acl [root@s3db1 ~]# rados -p " " stat .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 /.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 mtime 2021-03-10 06:27:43.000000, size 317 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 3

Many OSD marked down after no beacon for XXX seconds, just becauseone MON's OS disk was blocked.
by 912273695＠qq.com 10 Mar '21

10 Mar '21

Hello，everyone, In the OS disk blocked scene, the Mon service is still running ,but cant't work normally, soon the mon will out of quorum, but some OSD were still markd down after mon_osd_report_timeout*2 seconds, which will cause the cluster unavailable. At this time， the OS is very slow，may be unable to operate or login，so it‘s impossible to kill the mon by hand or other service. This problem exist in L and N version. Any suggestion？ you can reproduce this problem as follow: 1、 block the OS disk use follow command. # echo blocked > /sys/block/sdx/device/state 2、a few seconds later， the mon will out of quorum 3、after mon_osd_report_timeout*2 seconds，some osds will be markd down. （mon_osd_report_timeout default value 900s，you can set a smaller value） 912273695(a)qq.com

1 0

February 2021 Tech Talk and Code Walk-through
by Mike Perez 10 Mar '21

10 Mar '21

Hi everyone! I'm excited to announce two talks we have on the schedule for February 2021: Jason Dillaman will be giving part 2 to the librbd code walk-through. The stream starts on February 23rd at 18:00 UTC / 19:00 CET / 1:00 PM EST / 10:00 AM PST https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs Part 1: https://www.youtube.com/watch?v=L0x61HpREy4 -------------- What's New in the Pacific Release Hear Sage Weil give a live update on the development of the Pacific Release. The stream starts on February 25th at 17:00 UTC / 18:00 CET / 12 PM EST / 9 AM PST. https://ceph.io/ceph-tech-talks/ All live streams will be recorded and -- Mike Perez

1 2

Ceph User Survey Now Available
by Mike Perez 09 Mar '21

09 Mar '21

Hi everyone! Be sure to make your voice heard by taking the Ceph User Survey before April 2, 2021. This information will help guide the Ceph community’s investment in Ceph and the Ceph community's future development. https://ceph.io/user-survey/ Thank you to the Ceph User Survey Working Group for designing this year's survey! * Anantha Adiga * Anthony D'Atri * Paul Mezzanini * Stefan Kooman https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group We will provide the final results in the coming months after the survey has ended. -- Mike Perez

1 1

PG stuck at active+clean+remapped
by Michael Fladischer 09 Mar '21

09 Mar '21

Hi, we have replaced some of our OSDs a while ago an while everything recovery as planned, one PG is still stuck at active+clean+remapped with no backfilling taking place. Mpaaing the PG in question shows me that one OSD is missing: $ ceph pg map 35.1fe osdmap e1265760 pg 35.1fe (35.1fe) -> up [97,190,65,23,393,223,2147483647,354,132] acting [97,190,65,23,393,223,112,354,132] It seems that osd.112 should be replaced with an other OSD and I suspect that CRUSH cannot find a suitable one. Pool 35 is EC and has k=7 and m=2 and our Cluster has 9 OSD nodes. Is this just a case of CRUSH giving up to early as described in the troubleshooting PGs section[0] of the docs? Running the test as described there using `crushtool` gives several bad mapping rule results for "--num-rep 9". If so, would it help to just add new OSDs to the existing hosts or would it be better to add a whole new OSD host? Are there other options (upmap) to force this single PG to use a different set of OSDs for its "up" map? [0] https://github.com/ceph/ceph/blob/master/doc/rados/troubleshooting/troubles… Thanks, Michael

1 0

node down pg with backfill_wait waiting for incomplete?
by Marc 09 Mar '21

09 Mar '21

I have a node down and pg's are remapping/backfilling. I have also a lot of pg's in backfill_wait. I was wondering if there is a specific order that this is being executed. Eg I have a large'garbage' pool ec21 that is stuck. I could resolve that by changing the min size. However I rather have this not remapped, and wait for this node to be back on line. Question is: is other remapping/backfilling waiting for this stuck to be fixed, or is backfilling/remapping so smart that it will do what ever it can? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)12 4466 845 f: +48 (0)12 4466 843 e: marc(a)f1-outsourcing.eu

1 0

Rados gateway basic pools missing
by St-Germain, Sylvain (SSC/SPC) 09 Mar '21

09 Mar '21

Hi everyone, I just rebuild a (test) cluster using : OS : Ubuntu 20.04.2 LTS CEPH : ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable) 3 nodes : monitor/storage 1. The cluster looks good : # ceph -s cluster: id: 9a89aa5a-1702-4f87-a99c-f94c9f2cdabd health: HEALTH_OK services: mon: 3 daemons, quorum dao-wkr-04,dao-wkr-05,dao-wkr-06 (age 7m) mgr: dao-wkr-05(active, since 8m), standbys: dao-wkr-04, dao-wkr-06 mds: cephfs:1 {0=dao-wkr-04=up:active} 2 up:standby osd: 9 osds: 9 up (since 7m), 9 in (since 4h) rgw: 3 daemons active (dao-wkr-04.rgw0, dao-wkr-05.rgw0, dao-wkr-06.rgw0) task status: data: pools: 7 pools, 121 pgs objects: 234 objects, 16 KiB usage: 9.0 GiB used, 2.0 TiB / 2.0 TiB avail pgs: 121 active+clean 2. except that the main pools for the radosgw are not there # sudo ceph osd lspools 1 device_health_metrics 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.log 6 default.rgw.control 7 default.rgw.meta Missing : default.rgw.buckets.index & default.rgw.buckets.data What do you think ? Thx ! Sylvain

1 1

Replacing disk with xfs on it, documentation?
by Drew Weaver 09 Mar '21

09 Mar '21

Hello, I haven't needed to replace a disk in awhile and it seems that I have misplaced my quick little guide on how to do it. When searching the docs it is now recommending that you should use ceph-volume to create OSDs when doing that it creates LV: Disk /dev/sde: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/ceph--34b5a0a9--f84a--416f--8b74--fb1e05161f80-osd--block--4581caf4--eef0--42e1--b237--c114dfde3d15: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Cool but all of my other OSDs look like this and appear to just be XFS. Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: gpt Disk identifier: D0195044-10B5-4113-8210-A5CFCB9213A2 # Start End Size Type Name 1 2048 206847 100M Ceph OSD ceph data 2 206848 7812939742 3.7T unknown ceph block 35075 ? S< 0:00 [xfs-buf/sdd1] 35076 ? S< 0:00 [xfs-data/sdd1] 35077 ? S< 0:00 [xfs-conv/sdd1] 35078 ? S< 0:00 [xfs-cil/sdd1] 35079 ? S< 0:00 [xfs-reclaim/sdd] 35080 ? S< 0:00 [xfs-log/sdd1] 35082 ? S 0:00 [xfsaild/sdd1] I can't seem to find the instructions in order to create an OSD like the original ones anymore. I'm pretty sure that when this cluster was setup it was setup using the: ceph-deploy osd prepare ceph-deploy osd create ceph-deploy osd activate commands, but it seems as though the prepare and activate commands have since been removed from ceph-deploy, so I am really confused. =) Does anyone by chance have the instructions for replacing a failed drive when it meets the above criteria?

1 0

DocuBetter Meeting This Week -- 10 Mar 2021 1730 UTC
by John Zachary Dover 09 Mar '21

09 Mar '21

This week's meeting will focus on the ongoing rewrite of the cephadm documentation and making certain that the documentation addresses the rough edges in the Pacific release. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

Bluestore OSD crash with tcmalloc::allocate_full_cpp_throw_oom in multisite setup with PG_DAMAGED cluster error
by Szabo, Istvan (Agoda) 09 Mar '21

09 Mar '21

Hi, I have a 3 DC multisite setup. The replication is directional like HKG->SGP->US so the bucket is replicated from HKG to SGP and the same bucket is replicated further from SGP to US. The HKG > SGP connection is pretty fast 12.5millions objects (600GB) transferred in 6.5 hours. Once the OSD crashed in SGP, it stopped the complete chain replication and made PG_DAMAGED cluster error. The pg can be repaired but the sync never started back only bucket sync disable/enable helped. I got OSD crash also in HKG BUT in ASH. ASH no any error, the replication speed is 2 millions objects in 6.5 hours which is like 90GB of data. This is the crash of the osd: { "backtrace": [ "(()+0x12b20) [0x7f597d3fbb20]", "(gsignal()+0x10f) [0x7f597c0667ff]", "(abort()+0x127) [0x7f597c050c35]", "(()+0x9009b) [0x7f597ca1c09b]", "(()+0x9653c) [0x7f597ca2253c]", "(()+0x96597) [0x7f597ca22597]", "(()+0x967f8) [0x7f597ca227f8]", "(()+0x19d24) [0x7f597e168d24]", "(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7f597e18b0d6]", "(rocksdb::Arena::AllocateNewBlock(unsigned long)+0x43) [0x5632f08ccb93]", "(rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x4b) [0x5632f08ccc3b]", "(rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0xb4) [0x5632f07fae94]", "(()+0x1103e7e) [0x5632f085ae7e]", "(rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, bool, rocksdb::MemTablePostProcessInfo*)+0xcf) [0x5632f07f6f8f]", "(rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::ValueType)+0x452) [0x5632f08520e2]", "(rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&)+0x17) [0x5632f0852e97]", "(rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x480) [0x5632f084ac20]", "(rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, bool, bool, bool)+0x149) [0x5632f084ebe9]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1acd) [0x5632f078a03d]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x5632f078ac11]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x8c) [0x5632f074180c]", "(RocksDBStore::submit_transaction(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x87) [0x5632f0742027]", "(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x426) [0x5632f0226376]", "(BlueStore::_kv_sync_thread()+0x176f) [0x5632f024bc1f]", "(BlueStore::KVSyncThread::entry()+0x11) [0x5632f0273791]", "(()+0x814a) [0x7f597d3f114a]", "(clone()+0x43) [0x7f597c12bf23]" ], "ceph_version": "15.2.9", "crash_id": "2021-03-04T14:55:45.094048Z_3d481fd3-7573-4cb7-9b22-20784b418e64", "entity_name": "osd.5", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "9643c370a20c0d34f5e8965ae4461e2a7cf709ab4183929239bc263d0e1eef94", "timestamp": "2021-03-04T14:55:45.094048Z", "utsname_hostname": "hostname", "utsname_machine": "x86_64", "utsname_release": "4.18.0-240.10.1.el8_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Mon Jan 18 17:05:51 UTC 2021" } Any idea what I should tune? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

ny way can recover data?
by Elians Wan 09 Mar '21

09 Mar '21

we use s3 api upload file to ceph cluster with non versioned bucket, and we override many files then want to recover them, any way can recover?

1 0

2 Pgs (1x inconsistent, 1x unfound / degraded - unable to fix
by Jeremi Avenant 09 Mar '21

09 Mar '21

Good day I'm currently decommissioning a cluster that runs EC3+1 (rack failure domain - with 5 racks), however the cluster still has some production items on it since I'm in the process of moving it to our new EC8+2 cluster. Running Luminous 12.2.13 on Ubuntu 16 HWE, containerized with ceph-ansible 3.2. I currently get the following error after we lost 1 OSD (195). I'm forced to repair, scrub, deep scrub, restart OSDs etc, everything mentioned in the troubleshooting docs & information from IRC but cannot for the life of me get it to work. What I'm seeing is, that pg 9.3dd (volume_images) has a status of 1 OSD (osd is down) which I know, but the other OSD shows 316 (not queried). Also pg 9.3dd has 4x functioning UP's but still reference the missing OSD under acting. Regards OBJECT_UNFOUND 1/501815192 objects unfound (0.000%) pg 9.3dd has 1 unfound objects OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 9.3d1 is active+clean+inconsistent, acting [347,316,307,249] PG_DEGRADED Degraded data redundancy: 1219/2001837265 objects degraded (0.000%), 1 pg degraded, 1 pg undersized pg 9.3dd is stuck undersized for 55486.439002, current state active+recovery_wait+forced_recovery+undersized+degraded+remapped, last acting [355,2147483647,64,367] ceph pg 9.3dd query "up": [ 355, 139, 64, 367 ], "acting": [ 355, 2147483647, 64, 367 "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2021-03-08 16:07:51.239010", "might_have_unfound": [ { "osd": "64(2)", "status": "already probed" }, { "osd": "139(1)", "status": "already probed" }, { "osd": "195(1)", "status": "osd is down" }, { "osd": "316(2)", "status": "not queried" }, { "osd": "367(3)", "status": "already probed" } ], "recovery_progress": { "backfill_targets": [ "139(1)" ceph pg 9.3d1 query { "state": "active+clean+inconsistent", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 168488, "up": [ 347, 316, 307, 249 ], "acting": [ 347, 316, 307, 249 ], "actingbackfill": [ "249(3)", "307(2)", "316(1)", "347(0)" -- CLUSTER STATS: cluster: id: 1ea59fbe-46a4-474e-8225-a66b32ca86b7 health: HEALTH_ERR 1/490166525 objects unfound (0.000%) 1 scrub errors Possible data damage: 1 pg inconsistent Degraded data redundancy: 1259/1956055233 objects degraded (0.000%), 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum B-04-11-cephctl,B-05-11-cephctl,B-03-11-cephctl mgr: B-03-11-cephctl(active), standbys: B-04-11-cephctl, B-05-11-cephctl mds: cephfs-1/1/1 up {0=B-04-11-cephctl=up:active}, 2 up:standby osd: 384 osds: 383 up, 383 in; 1 remapped pgs data: pools: 11 pools, 13440 pgs objects: 490.17M objects, 1.35PiB usage: 1.88PiB used, 2.33PiB / 4.21PiB avail pgs: 1259/1956055233 objects degraded (0.000%) 1/490166525 objects unfound (0.000%) 13332 active+clean 96 active+clean+scrubbing+deep 10 active+clean+scrubbing 1 active+clean+inconsistent 1 active+recovery_wait+forced_recovery+undersized+degraded+remapped *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

1 0

Any way can recover data?
by Elians Wan 09 Mar '21

09 Mar '21

we use s3 api upload file to ceph cluster with non versioned bucket, and we override many files then want to recover them, any way can recover?

1 0

Unable to delete bucket - endless multipart uploads?
by David Monschein 08 Mar '21

08 Mar '21

Hi All, We've been dealing with what seems to be a pretty annoying bug for a while now. We are unable to delete a customer's bucket that seems to have an extremely large number of aborted multipart uploads. I've had $(radosgw-admin bucket rm --bucket=pusulax --purge-objects) running in a screen session for almost 3 weeks now and it's still not finished; it's most likely stuck in a loop or something. The screen session with debug-rgw=10 spams billions of these messages: 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/d3/04d33e18-3f13-433c-b924-56602d702d60-31.msg.2~0DTalUjTHsnIiKraN1klwIFO88Vc2E3.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/d7/04d7ad26-c8ec-4a39-9938-329acd6d9da7-102.msg.2~K_gAeTpfEongNvaOMNa0IFwSGPpQ1iA.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/da/04da4147-c949-4c3a-aca6-e63298f5ff62-102.msg.2~-hXBSFcjQKbMkiyEqSgLaXMm75qFzEp.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/db/04dbb0e6-dfb0-42fb-9d0f-49cceb18457f-102.msg.2~B5EhGgBU5U_U7EA5r8IhVpO3Aj2OvKg.meta[] 2021-02-23 15:38:58.667 7f9b55704840 10 RGWRados::cls_bucket_list_unordered: got _multipart_04/df/04df39be-06ab-4c72-bc63-3fac1d2700a9-11.msg.2~_8h5fWlkNrIMqcrZgNbAoJfc8BN1Xx-.meta[] This is probably the 2nd or 3rd time I've been unable to delete this bucket. I also tried running $(radosgw-admin bucket check --fix --check-objects --bucket=pusulax) before kicking off the delete job, but that didn't work either. Here is the bucket in question, the num_objects counter never decreases after trying to delete the bucket: [root@os5 ~]# radosgw-admin bucket stats --bucket=pusulax { "bucket": "pusulax", "num_shards": 144, "tenant": "", "zonegroup": "dbb69c5b-b33f-4af2-950c-173d695a4d2c", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "c0d0b8a5-c63c-4c24-9dab-8deee88dbf0b.3209338.4", "marker": "c0d0b8a5-c63c-4c24-9dab-8deee88dbf0b.3292800.7", "index_type": "Normal", "owner": "REDACTED", "ver": "0#115613,1#115196,2#115884,3#115497,4#114649,5#114150,6#116127,7#114269,8#115220,9#115092,10#114003,11#114538,12#115235,13#113463,14#114928,15#115135,16#115535,17#114867,18#116010,19#115766,20#115274,21#114818,22#114805,23#114853,24#114099,25#114359,26#114966,27#115790,28#114572,29#114826,30#114767,31#115614,32#113995,33#115305,34#114227,35#114342,36#114144,37#114704,38#114088,39#114738,40#114133,41#114520,42#114420,43#114168,44#113820,45#115093,46#114788,47#115522,48#114713,49#115315,50#115055,51#114513,52#114086,53#114401,54#114079,55#113649,56#114089,57#114157,58#114064,59#115224,60#114753,61#114686,62#115169,63#114321,64#114949,65#115075,66#115003,67#114993,68#115320,69#114392,70#114893,71#114219,72#114190,73#114868,74#113432,75#114882,76#115300,77#114755,78#114598,79#114221,80#114895,81#114031,82#114566,83#113849,84#115155,85#113790,86#113334,87#113800,88#114856,89#114841,90#115073,91#113849,92#114554,93#114820,94#114256,95#113840,96#114838,97#113784,98#114876,99#115524,100#115686,101#112969,102#112156,103#112635,104#112732,105#112933,106#112412,107#113090,108#112239,109#112697,110#113444,111#111730,112#112446,113#114479,114#113318,115#113032,116#112048,117#112404,118#114545,119#112563,120#112341,121#112518,122#111719,123#112273,124#112014,125#112979,126#112209,127#112830,128#113186,129#112944,130#111991,131#112865,132#112688,133#113819,134#112586,135#113275,136#112172,137#113019,138#112872,139#113130,140#112716,141#112091,142#111859,143#112773", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0", "mtime": "2020-06-17 20:27:16.685833Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#", "usage": { "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 97009704, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 94737, "num_objects": 3592952 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } We're running 14.2.16 on our RGWs and OSD nodes. Anyone have any ideas? Is it possible to target this bucket via rados directly to try and delete it? I'm weary of doing stuff like that though. Thanks in advance. - Dave Monschein

3 2

Re: Ceph Object Gateway setup/tutorial
by Juan Miguel Olmo Martinez 08 Mar '21

08 Mar '21

Hi Rok Using cephadm you do not need to worry about changes in the config files, restarts and this kind of things: Understand the RGW functionality, the link you provided is ok. ( https://docs.ceph.com/en/latest/radosgw/ ) but for installation, and management of the RGW daemons use: https://docs.ceph.com/en/latest/cephadm/rgw/ -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

1 0

Questions RE: Ceph/CentOS/IBM
by Drew Weaver 08 Mar '21

08 Mar '21

Howdy, After the IBM acquisition of RedHat the landscape for CentOS quickly changed. As I understand it right now Ceph 14 is the last version that will run on CentOS/EL7 but CentOS8 was "killed off". So given that, if you were going to build a Ceph cluster today would you even bother doing it using a non-commercial distribution or would you just use RHEL 8 (or even their commercial Ceph product). Secondly, are we expecting IBM to "kill off" Ceph as well? Thanks, -Drew

15 20

Can't clear UPGRADE_REDEPLOY_DAEMON after fix
by Samy Ascha 08 Mar '21

08 Mar '21

Hi! I was upgrading from 15.2.8 to 15.2.9 via `ceph orch upgrade` (Ubuntu Bionic). One OSD seemed to have failed to upgrade, so I just redeployed it. The OSD is up/in, but this warning is not clearing: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.3 on host ceph-osd4 failed. It seems the warning is not listed in the docs, so I'm wondering how to go about getting the cluster back into HEALTH_OK. Hope you can help, Samy

2 2

balance OSD usage.
by ricardo.re.azevedo＠gmail.com 08 Mar '21

08 Mar '21

Hi All, Does anyone know how I can rebalance my cluster to balance out the OSD usage? I just added 12 more 14Tb HDDs to my cluster (cluster of made up of 12Tb and 14Tb disks) bringing my total to 48 OSDs. Ceph df reports my pool as 83% full (see below). I am aware this only reports the fullest drive. However, after backfilling, data distribution is still heavily skewed to a few OSDs. The oldest are at ~80% and the newest are at ~50. My issue is that data is currently being written to all drives equally and at this rate the cluster will start disabling writes despite still having significant raw capacity. PGs do not seem to be evenly distributed either. Weights are 12.73 for all 14Tb OSD and 10.9 for all 12Tb OSDs. I tried using the balancer to create a new plan but it had no real effect. Haven't seen much useful info online. --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 582 TiB 203 TiB 379 TiB 380 TiB 65.20 TOTAL 582 TiB 203 TiB 379 TiB 380 TiB 65.20 --- POOLS --- POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 2 261 TiB 68.62M 379 TiB 83.64 56 TiB cephfs_metadata 3 536 MiB 12.74k 1.6 GiB 0 25 TiB `osd df`: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 10.91409 1.00000 11 TiB 8.4 TiB 8.4 TiB 1.6 MiB 22 GiB 2.5 TiB 77.00 1.18 92 up 4 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 21 MiB 18 GiB 4.0 TiB 63.03 0.97 80 up 5 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.3 MiB 17 GiB 4.4 TiB 59.50 0.91 71 up 11 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.4 TiB 36 MiB 24 GiB 3.3 TiB 74.13 1.14 105 up 15 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.2 TiB 20 MiB 21 GiB 4.5 TiB 64.75 0.99 99 up 16 hdd 10.91409 1.00000 11 TiB 6.4 TiB 6.4 TiB 1.7 MiB 16 GiB 4.5 TiB 58.81 0.90 74 up 20 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 33 MiB 23 GiB 4.3 TiB 66.09 1.01 102 up 22 hdd 12.73340 1.00000 13 TiB 8.6 TiB 8.6 TiB 3.3 MiB 22 GiB 4.1 TiB 67.48 1.04 100 up 23 hdd 12.73340 1.00000 13 TiB 7.6 TiB 7.6 TiB 406 KiB 20 GiB 5.1 TiB 59.93 0.92 89 up 36 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.2 TiB 171 KiB 16 GiB 5.5 TiB 57.00 0.87 85 up 41 hdd 12.73340 1.00000 13 TiB 7.0 TiB 7.0 TiB 2.1 MiB 16 GiB 5.7 TiB 54.85 0.84 85 up 46 hdd 12.73340 1.00000 13 TiB 8.1 TiB 8.0 TiB 67 MiB 19 GiB 4.7 TiB 63.34 0.97 99 up 3 hdd 10.91409 1.00000 11 TiB 7.6 TiB 7.6 TiB 1.0 MiB 19 GiB 3.3 TiB 69.85 1.07 87 up 6 hdd 10.91409 1.00000 11 TiB 8.3 TiB 8.3 TiB 39 MiB 22 GiB 2.6 TiB 76.28 1.17 93 up 10 hdd 12.73340 1.00000 13 TiB 9.6 TiB 9.6 TiB 5.1 MiB 24 GiB 3.1 TiB 75.52 1.16 109 up 14 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.8 TiB 34 MiB 22 GiB 3.9 TiB 69.57 1.07 99 up 24 hdd 12.73340 1.00000 13 TiB 11 TiB 11 TiB 22 MiB 34 GiB 2.1 TiB 83.28 1.28 117 up 25 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 24 MiB 21 GiB 4.4 TiB 65.84 1.01 97 up 26 hdd 12.73340 1.00000 13 TiB 8.9 TiB 8.9 TiB 205 KiB 22 GiB 3.8 TiB 69.84 1.07 102 up 30 hdd 10.91409 1.00000 11 TiB 6.9 TiB 6.9 TiB 33 MiB 16 GiB 4.0 TiB 63.31 0.97 92 up 39 hdd 12.73340 1.00000 13 TiB 7.2 TiB 7.2 TiB 36 MiB 16 GiB 5.6 TiB 56.39 0.86 93 up 43 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 3.6 MiB 19 GiB 4.4 TiB 65.54 1.01 98 up 47 hdd 12.73340 1.00000 13 TiB 7.1 TiB 7.1 TiB 51 MiB 17 GiB 5.6 TiB 55.70 0.85 88 up 2 hdd 10.91409 1.00000 11 TiB 7.4 TiB 7.3 TiB 22 KiB 19 GiB 3.6 TiB 67.36 1.03 82 up 7 hdd 10.91409 1.00000 11 TiB 8.5 TiB 8.5 TiB 530 KiB 21 GiB 2.4 TiB 77.89 1.19 97 up 8 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 21 MiB 22 GiB 4.3 TiB 66.53 1.02 99 up 12 hdd 12.73340 1.00000 13 TiB 8.5 TiB 8.5 TiB 20 MiB 22 GiB 4.2 TiB 67.13 1.03 99 up 18 hdd 12.73340 1.00000 13 TiB 9.9 TiB 9.9 TiB 24 MiB 31 GiB 2.8 TiB 77.81 1.19 112 up 19 hdd 12.73340 1.00000 13 TiB 8.8 TiB 8.8 TiB 34 MiB 22 GiB 3.9 TiB 69.17 1.06 104 up 21 hdd 12.73340 1.00000 13 TiB 9.5 TiB 9.4 TiB 15 MiB 28 GiB 3.3 TiB 74.37 1.14 111 up 31 hdd 10.91409 1.00000 11 TiB 6.3 TiB 6.3 TiB 56 MiB 14 GiB 4.6 TiB 57.53 0.88 77 up 32 hdd 10.91409 1.00000 11 TiB 8.0 TiB 8.0 TiB 44 MiB 18 GiB 2.9 TiB 73.15 1.12 99 up 37 hdd 12.73340 1.00000 13 TiB 8.2 TiB 8.1 TiB 4.2 MiB 18 GiB 4.6 TiB 64.09 0.98 94 up 40 hdd 12.73340 1.00000 13 TiB 7.3 TiB 7.3 TiB 1.4 MiB 16 GiB 5.5 TiB 57.10 0.88 86 up 44 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 2.6 MiB 17 GiB 5.0 TiB 60.55 0.93 92 up 1 hdd 10.91409 1.00000 11 TiB 5.9 TiB 5.9 TiB 55 MiB 15 GiB 5.0 TiB 54.14 0.83 72 up 9 hdd 12.73340 1.00000 13 TiB 9.4 TiB 9.3 TiB 705 KiB 24 GiB 3.4 TiB 73.57 1.13 109 up 13 hdd 12.73340 1.00000 13 TiB 8.0 TiB 8.0 TiB 22 MiB 21 GiB 4.7 TiB 62.74 0.96 91 up 17 hdd 10.91409 1.00000 11 TiB 7.5 TiB 7.5 TiB 24 MiB 19 GiB 3.4 TiB 68.88 1.06 88 up 27 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.8 TiB 4.8 MiB 17 GiB 5.9 TiB 53.81 0.83 78 up 28 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 19 MiB 20 GiB 5.0 TiB 60.46 0.93 91 up 29 hdd 12.73340 1.00000 13 TiB 8.3 TiB 8.3 TiB 17 MiB 21 GiB 4.4 TiB 65.56 1.01 100 up 33 hdd 10.91409 1.00000 11 TiB 6.5 TiB 6.5 TiB 1.6 MiB 15 GiB 4.4 TiB 59.23 0.91 77 up 34 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.6 TiB 44 MiB 15 GiB 4.3 TiB 60.87 0.93 81 up 35 hdd 10.91409 1.00000 11 TiB 6.6 TiB 6.5 TiB 18 MiB 15 GiB 4.4 TiB 60.08 0.92 81 up 38 hdd 12.73340 1.00000 13 TiB 8.4 TiB 8.4 TiB 18 MiB 19 GiB 4.3 TiB 66.25 1.02 105 up 42 hdd 12.73340 1.00000 13 TiB 6.9 TiB 6.9 TiB 21 MiB 16 GiB 5.8 TiB 54.26 0.83 84 up 45 hdd 12.73340 1.00000 13 TiB 7.7 TiB 7.7 TiB 38 MiB 18 GiB 5.0 TiB 60.61 0.93 94 up TOTAL 582 TiB 380 TiB 379 TiB 942 MiB 950 GiB 203 TiB 65.20 Thank you, Ricardo

4 3

Re: Resolving LARGE_OMAP_OBJECTS
by Benoît Knecht 08 Mar '21

08 Mar '21

Hi Drew, On Friday, March 5th, 2021 at 20:22, Drew Weaver <drew.weaver(a)thenap.com> wrote: > Sorry for multi-reply, I got that command to run: > > for obj in $(rados -p default.rgw.buckets.index ls | grep 2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637); do printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys $obj | wc -l); done; > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.0 10423 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.15 10445 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.3 10542 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.6 10511 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.13 10414 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.12 10479 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.2 10486 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.5 10448 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.4 10470 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.8 10474 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.1 10470 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.10 10411 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.7 10445 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.14 10413 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.9 10356 > .dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637.11 10410 So this looks fine, it means that this bucket is sharded properly and the index keys are evenly distributed. However, the problematic object is `.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.213.0`; you said there's no bucket with that ID, which means it's probably a stale instance. Does it still exist in `default.rgw.buckets.index`? If so, you can try running `radosgw-admin reshard stale-instances rm`, it should get rid of it (you can verify afterwards with `rados -p default.rgw.buckets.index ls`). -- Ben

1 0

grafana-api-url not only for one host
by E Taka 05 Mar '21

05 Mar '21

Hi, if the host fails, to which the grafana-api-url points (in the example below ceph01.hostxyz.tld:3000), Ceph Dashboard can't Display Grafana Data: # ceph dashboard get-grafana-api-url https://ceph01.hostxyz.tld:3000 Is it possible to automagically switch to an other host? Thanks, Erich

3 2

what is quickest way to generate a new key for a user?
by Marc 05 Mar '21

05 Mar '21

what is quickest way to generate a new key for a user?

1 0

Best practices for OSD on bcache
by Norman.Kern 04 Mar '21

04 Mar '21

Hi, guys I am testing ceph on bcache devices, I found the performance is not good as expected. Does anyone have any best practices for it? Thanks.

5 11

Metadata for LibRADOS
by Cary FitzHugh 04 Mar '21

04 Mar '21

Hello - We're trying to use native libRADOS and the only challenge we're running into is searching metadata. Using the rgw metadata sync seems to require all data to be pushed through the rgw, which is not something we're interested in setting up at the moment. Are there hooks or features of libRADOS which could be leveraged to enable syncing of metadata to an external system (elastic-search / postgres / etc)? Is there a way to listen to a stream of updates to a pool in real-time, with some guarantees I wouldn't miss things? Are there any features like this in libRADOS? Thank you

5 7

Re: Questions RE: Ceph/CentOS/IBM
by Marc 04 Mar '21

04 Mar '21

> This is wrong. Ceph 15 runs on CentOS 7 just fine, but without the > dashboard. > I also hope that ceph is keeping support for el7 till it is eol in 2024. So I have enough time to figure out what OS to choose.

2 2

CephFS: side effects of not using ceph-mgr volumes / subvolumes
by Sebastian Knust 03 Mar '21

03 Mar '21

Hi, Assuming a cluster (currently octopus, might upgrade to pacific once released) serving only CephFS and that only to a handful of kernel and fuse-clients (no OpenStack, CSI or similar): Are there any side effects of not using the ceph-mgr volumes module abstractions [1], namely subvolumes and subvolume groups, that I have to consider? I would still only mount subtrees of the whole (single) CephFS file system and have some clients which mount multiple disjunct subtrees. Quotas would only be set on the subtree level which I am mounting, likewise file layouts. Snapshots (via mkdir in .snap) would be used on the mounting level or one level above. Background: I don't require the abstraction features per se. Some restrictions (e.g. subvolume group snapshots not being supported) seem to me to be caused only by the abstraction layer and not the underlying CephFS. For my specific use case I require snapshots on the subvolume group layer. It therefore seems better to just forego the abstraction as a whole and work on bare CephFS. Cheers Sebastian [1] https://docs.ceph.com/en/octopus/cephfs/fs-volumes/

2 1

cephfs: unable to mount share with 5.11 mainline, ceph 15.2.9, MDS 14.1.16
by Stefan Kooman 03 Mar '21

03 Mar '21

Hi, On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with Ceph Octopus 15.2.9 packages installed. The MDS server is running Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the monitors is reachable from the client. At mount time we get the following: > Mar 2 09:01:14 kernel: Key type ceph registered > Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24) > Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching > Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32) > Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established > Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec > Mar 2 09:01:14 kernel: ceph: corrupt mdsmap > Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22 > Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec > Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f) > Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2 changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still seems to be use the v1 port though (although since 5.11 v2 should be supported). Has anyone seen this before? Just guessing here, but could it that the client tries to speak v2 protocol on v1 port? Thanks, Stefan

3 14

Monitor leveldb growing without bound v14.2.16
by Lincoln Bryant 03 Mar '21

03 Mar '21

Hi list, We recently had a cluster outage over the weekend where several OSDs were inaccessible over night for several hours. When I found the cluster in the morning, the monitors' root disks (which contained both the monitor's leveldb and the Ceph logs) had completely filled. After restarting OSDs, cleaning out the monitors' logs, moving /var/lib/ceph to dedicated disks on the mons, and starting recovery (in which there was 1 unfound object that I marked lost, if that has any relevancy), the leveldb continued/continues to grow without bound. The cluster has all PGs in active+clean at this point, yet I'm accumulating what seems like approximately ~1GB/hr of new leveldb data. Two of the monitors (a, c) are in quorum, while the third (b) has been synchronizing for the last several hours, but doesn't seem to be able to catch up. Mon 'b' has been running for 4 hours now in the 'synchronizing' state. The mon's log has many messages about compacting and deleting files, yet we never exit the synchronization state. The ceph.log is also rapidly accumulating complaints that the mons are slow (not surprising, I suppose, since the levelDBs are ~100GB at this point). I've found that using monstore tool to do compaction on mons 'a' and 'c' thelps but is only a temporary fix. Soon the database inflates again and I'm back to where I started. Thoughts on how to proceed here? Some ideas I had: - Would it help to add some new monitors that use RocksDB? - Stop a monitor and dump the keys via monstoretool, just to get an idea of what's going on? - Increase mon_sync_max_payload_size to try to move data in larger chunks? - Drop down to a single monitor, and see if normal compaction triggers and stops growing unbounded? - Stop both 'a' and 'c', compact them, start them, and immediately start 'b' ? Appreciate any advice. Regards, Lincoln

4 4

"optimal" tunables on release upgrade
by Matthew Vernon 03 Mar '21

03 Mar '21

Hi, Having been slightly caught out by tunables on my Octopus upgrade[0], can I just check that if I do ceph osd crush tunables optimal That will update the tunables on the cluster to the current "optimal" values (and move a lot of data around), but that this doesn't mean they'll change next time I upgrade the cluster or anything like that? It's not quite clear from the documentation whether the next time "optimal" tunables change that'll be applied to a cluster where I've set tunables thus, or if tunables are only ever changed by a fresh invocation of ceph osd crush tunables... [I assume the same answer applies to "default"?] Regards, Matthew [0] I foolishly thought a cluster initially installed as Jewel would have jewel tunables -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

2 1

Octopus auto-scale causing HEALTH_WARN re object numbers
by Matthew Vernon 03 Mar '21

03 Mar '21

Hi, I've upgraded our test cluster to Octopus, and enabled the auto-scaler. It's nearly finished: PG autoscaler decreasing pool 11 PGs from 1024 to 32 (4d) [==========================..] (remaining: 3h) But I notice it looks to be making pool 11 smaller when HEALTH_WARN thinks it should be larger: root@sto-t1-1:~# ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average; 9 pgs not deep-scrubbed in time [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than average pool default.rgw.buckets.data objects per pg (313153) is more than 23.4063 times cluster average (13379) ...which seems like the wrong thing for the auto-scaler to be doing. Is this a known problem? Regards, Matthew More details: ceph df: root@sto-t1-1:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 993 TiB 782 TiB 210 TiB 211 TiB 21.22 TOTAL 993 TiB 782 TiB 210 TiB 211 TiB 21.22 --- POOLS --- POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 2 69 KiB 4 1.4 MiB 0 220 TiB default.rgw.control 3 1.1 MiB 8 3.3 MiB 0 220 TiB default.rgw.data.root 4 115 KiB 14 3.6 MiB 0 220 TiB default.rgw.gc 5 5.3 MiB 32 23 MiB 0 220 TiB default.rgw.log 6 31 MiB 184 96 MiB 0 220 TiB default.rgw.users.uid 7 249 KiB 8 1.8 MiB 0 220 TiB default.rgw.buckets.data 11 23 GiB 10.02M 2.0 TiB 0.30 220 TiB rgwtls 13 54 KiB 3 843 KiB 0 220 TiB pilot-metrics 14 285 MiB 2.60M 476 GiB 0.07 220 TiB pilot-images 15 40 GiB 4.97k 122 GiB 0.02 220 TiB pilot-volumes 16 192 GiB 48.90k 577 GiB 0.09 220 TiB pilot-vms 17 125 GiB 33.79k 376 GiB 0.06 220 TiB default.rgw.users.keys 18 111 KiB 5 1.5 MiB 0 220 TiB default.rgw.buckets.index 19 4.0 GiB 246 12 GiB 0 220 TiB rbd 20 39 TiB 10.09M 116 TiB 14.88 220 TiB default.rgw.buckets.non-ec 21 344 KiB 1 1.0 MiB 0 220 TiB rgw-ec 22 7.0 TiB 1.93M 11 TiB 1.57 441 TiB rbd-ec 23 45 TiB 11.73M 67 TiB 9.22 441 TiB default.rgw.users.email 24 23 MiB 1 69 MiB 0 220 TiB pilot-backups 25 73 MiB 3 219 MiB 0 220 TiB device_health_metrics 26 51 MiB 186 153 MiB 0 220 TiB root@sto-t1-1:~# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE .rgw.root 70843 3.0 992.7T 0.0000 1.0 32 on default.rgw.control 1116k 3.0 992.7T 0.0000 1.0 32 on default.rgw.data.root 115.1k 3.0 992.7T 0.0000 1.0 32 on default.rgw.gc 5379k 3.0 992.7T 0.0000 1.0 32 on default.rgw.log 32036k 3.0 992.7T 0.0000 1.0 32 on default.rgw.users.uid 248.7k 3.0 992.7T 0.0000 1.0 32 on default.rgw.buckets.data 23894M 3.0 992.7T 0.0001 1.0 32 on rgwtls 55760 3.0 992.7T 0.0000 1.0 32 on pilot-metrics 285.3M 3.0 992.7T 0.0000 1.0 32 on pilot-images 41471M 3.0 992.7T 0.0001 1.0 32 on pilot-volumes 192.3G 3.0 992.7T 0.0006 1.0 32 on pilot-vms 124.6G 3.0 992.7T 0.0004 1.0 32 on default.rgw.users.keys 111.1k 3.0 992.7T 0.0000 1.0 32 on default.rgw.buckets.index 4090M 3.0 992.7T 0.0000 1.0 32 on rbd 39430G 3.0 992.7T 0.1164 1.0 1024 on default.rgw.buckets.non-ec 344.3k 3.0 992.7T 0.0000 1.0 32 on rgw-ec 7175G 1.5 992.7T 0.0106 1.0 64 on rbd-ec 45806G 1.5 992.7T 0.0676 1.0 1024 on default.rgw.users.email 23530k 3.0 992.7T 0.0000 1.0 32 on pilot-backups 74699k 3.0 992.7T 0.0000 1.0 32 on device_health_metrics 52128k 3.0 992.7T 0.0000 1.0 32 on -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

1 1

bug in latest cephadm bootstrap: got an unexpected keyword argument 'verbose_on_failure'
by Philip Brown 03 Mar '21

03 Mar '21

Seems like someone is not testing cephadm on centos 7.9 Just tried installing cephadm from the repo, and ran cephadm bootstrap --mon-ip=xxx it blew up, with ceph TypeError: __init__() got an unexpected keyword argument 'verbose_on_failure' just after the firewall section. I happen to have a test cluser from a few months ago, and compared the code. Some added, in line 2348, " out, err, ret = call([self.cmd, '--permanent', '--query-port', tcp_port], verbose_on_failure=False)" this made the init fail, on my centos 7.9 system, freshly installed and updated today. # cephadm version ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable) Simply commenting out that line makes it complete the cluster init like I remember. -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 1

Multiple radosgw on the same server
by Szabo, Istvan (Agoda) 03 Mar '21

03 Mar '21

Hi, I've heard many time that to install multiple rados-gateway on the same server is possible, just need to create on a different port. However I've never managed to make it work. Today I gave another try like this: 1. Created a new keyring: ceph auth get-or-create client.rgw.servername.rgw1 mon 'allow rw' osd 'allow rwx' 2. Created the keyring file: /var/lib/ceph/radosgw/ceph-rgw.servername.rgw1/keyring 3. Added another entry in the ceph octopus configuration with different port: [client.rgw.servername.rgw1] host = servername keyring = /var/lib/ceph/radosgw/ceph-rgw.servername.rgw1/keyring log file = /var/log/ceph/ceph-rgw-servername.rgw1.log rgw frontends = beast endpoint=10.104.198.101:8081 rgw thread pool size = 512 rgw_zone=zone 1. Copied another RGW system file in centos 8: cp -pr /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw\(a)rgw.servername.rgw0.service /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw\(a)rgw.servername.rgw1.service 2. Restarted ceph.target. 3. Result is the same number of rados-gateways. So how is this actually should have been done? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Need Clarification on Maintenance Shutdown Procedure
by Dave Hall 02 Mar '21

02 Mar '21

Hello, I've had a look at the instructions for clean shutdown given at https://ceph.io/planet/how-to-do-a-ceph-cluster-maintenance-shutdown/, but I'm not clear about some things on the steps about shutting down the various Ceph components. For my current 3-node cluster I have MONs, MDSs, MGRs, and OSDs all running on the same nodes. Also, this is a non-container installation. Since I don't have separate dedicated nodes, as described in the referenced web page, I think the instructions mean that I need to issue SystemD commands to stop the corresponding services/targets on each node for the Ceph components mentioned in each step. Since we want to bring services up in the right order, I should also use SystemD commands to disable these services/targets so they don't automatically restart when I power the nodes back on. After power-on, I would then re-enable and manually start services/targets in the order described. One other specific question: For step 4 it says to shut down my service nodes. Does this mean my MDSs? (I'm not running any Object Gateways or NFS, but I think these would go in this step as well?) Please let me know if I've got this right. The cluster contains 200TB of a researcher's data that has taken a year to collect, so caution is needed. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

3 3

Remapped PGs
by David Orman 02 Mar '21

02 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

MDS is reporting damaged metadata damage- followup
by ricardo.re.azevedo＠gmail.com 02 Mar '21

02 Mar '21

Hi all, Following up on a previous issue. My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace". ` ceph tell mds.database-0 scrub start / recursive repair` seems to have no effect on the damage. ` ceph tell mds.database-0 scrub start / recursive repair force` also has no effect. I understand this seems to be an issue with mapping the file to a filesystem path. Is there anything I can do to recover these files? Any manual methods? > ceph status reports: cluster: id: 692905c0-f271-4cd8-9e43-1c32ef8abd13 health: HEALTH_ERR 1 MDSs report damaged metadata 300 pgs not deep-scrubbed in time 300 pgs not scrubbed in time services: mon: 3 daemons, quorum database-0,file-server,webhost (age 37m) mgr: webhost(active, since 3d), standbys: file-server, database-0 mds: cephfs:1 {0=database-0=up:active} 2 up:standby osd: 48 osds: 48 up (since 56m), 48 in (since 13d); 10 remapped pgs task status: scrub status: mds.database-0: idle data: pools: 7 pools, 633 pgs objects: 60.82M objects, 231 TiB usage: 336 TiB used, 246 TiB / 582 TiB avail pgs: 623 active+clean 6 active+remapped+backfilling 4 active+remapped+backfill_wait Thanks for the help. Best, Ricardo

1 0

Ceph Object Gateway setup/tutorial
by Rok Jaklič 02 Mar '21

02 Mar '21

Hi, installation of cluster/osds went "by the book" https://docs.ceph.com/, but now I want to setup Ceph Object Gateway, but documentation on https://docs.ceph.com/en/latest/radosgw/ seems to lack information about what and where to restart for example when setting [client.rgw.gateway-node1] in /etc/ceph/ceph.conf. Also where should we set this? In cephadm shell or on the host ...? Is there some tutorial howto setup gateway from the beginning? Kind regards, Rok

1 0

reboot breaks OSDs converted from ceph-disk to ceph-volume simple
by Frank Schilder 02 Mar '21

02 Mar '21

Dear all, ceph version: mimic 13.2.10 I'm facing a serious bug with devices converted from "ceph-disk" to "ceph-volume simple". I "converted" all ceph-disk devices using "ceph-volume simple scan ..." And everything worked fine at the beginning. Today I needed to reboot an OSD host and since then most ceph-disk OSDs are screwed up. Apparently, "ceph-volume simple scan ..." creates symlinks to the block partition /dev/sd?2 using the "/dev/sd?2" name for the link target. These names are not stable and are expected to change after every reboot. Now I have a bunch of OSDs with new /dev/sd?2" names that won't boot any more, because this link points to the wrong block partition. Doing another "ceph-volume simple scan ..." doesn't help, it just "rediscovers" the wrong location. Here is what a broken OSD looks like (fresh "ceph-volume simple scan --stdout ..." output): { "active": "ok", "block": { "path": "/dev/sda2", "uuid": "b5ac1462-510a-4483-8f42-604e6adc5c9d" }, "block_uuid": "1d9d89a2-18c7-4610-9dcd-167d44ce1879", "bluefs": 1, "ceph_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "cluster_name": "ceph", "data": { "path": "/dev/sdb1", "uuid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb" }, "fsid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb", "keyring": "AQAZJ6ddedALDxAAJI7NLJ2CRFoQWK5STRpHuw==", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "none": "", "ready": "ready", "require_osd_release": "", "type": "bluestore", "whoami": 241 } OSD 241's data partition looks like this (after mount /dev/sdb1 /var/lib/ceph/osd/ceph-241): [root@ceph-adm:ceph-18 ceph-241]# ls -l /var/lib/ceph/osd/ceph-241 total 56 -rw-r--r--. 1 root root 411 Oct 16 2019 activate.monmap -rw-r--r--. 1 ceph ceph 3 Oct 16 2019 active lrwxrwxrwx. 1 root root 9 Mar 2 14:19 block -> /dev/sda2 -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 block_uuid -rw-r--r--. 1 ceph disk 2 Oct 16 2019 bluefs -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 ceph_fsid -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 fsid -rw-------. 1 ceph ceph 58 Oct 16 2019 keyring -rw-r--r--. 1 ceph disk 8 Oct 16 2019 kv_backend -rw-r--r--. 1 ceph ceph 21 Oct 16 2019 magic -rw-r--r--. 1 ceph disk 4 Oct 16 2019 mkfs_done -rw-r--r--. 1 ceph ceph 0 Nov 23 14:58 none -rw-r--r--. 1 ceph disk 6 Oct 16 2019 ready -rw-r--r--. 1 ceph disk 2 Jan 31 2020 require_osd_release -rw-r--r--. 1 ceph ceph 10 Oct 16 2019 type -rw-r--r--. 1 ceph ceph 4 Oct 16 2019 whoami The symlink "block -> /dev/sda2" goes to the wrong disk. How can I fix that in a stable way? Also, why are not stable "/dev/disk/by-uuid/..." link targets created instead? Can I change that myself? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

RBD clone on Bluestore
by Pawel S 02 Mar '21

02 Mar '21

hello! I'm trying to understand how Bluestore cooperates with RBD image clones, so my test is simple 1. create an image (2G) and fill with data 2. create a snapshot 3. protect it 4. create a clone of the image 5. write a small portion of data (4K) to clone 6. check how it changed and if just 4K are used to prove CoW allocated new extent instead of copying out snapped data. Unfortunately it occurs that at least rbd du reports that 4M was changed and the clone consumes 4M of data instead of expected 4K... ''' rbd du rbd/clone1 NAME PROVISIONED USED clone1 2 GiB 4 MiB ''' How can I trace/prove Bluestore CoW really works in this case, and prevent copying the rest of the 4M stripe like Filestore did ? p.s tested on Luminous/Octopus, ssd devices, min_alloc_size: 16k, block_size: 4k best regards! -- Pawel S.

3 4

Ceph iscsi help
by Várkonyi János 01 Mar '21

01 Mar '21

Hi All, I2d like to install a Ceph Nautilus on Ubuntu 18.04 LTS and give the storage to 2 windows server via ISCSI. I choose the Nautilus because of the deploy function I don't want to another VM to cephadm. So I can isntall the ceph and it is working properly but can't setup the icsi gateway. The services running like tcmu-runner, ebd-target-gw and rbd-target-api. I can going into gwcli but I can't create the first gw I get this msessage: /iscsi-target...-igw/gateways> create cf01 192.168.203.51 skipchecks=true OS version/package checks have been bypassed Get gateway hostname failed : 403 Forbidden Please check api_host setting and make sure host cf01 IP is listening on port 5000 In the syslog at the same time: Mar 1 15:43:02 cf01 there is no tcmu-runner data avaliable Mar 1 15:43:06 cf01 ::ffff:127.0.0.1 - - [01/Mar/2021 15:43:06] "GET /api/config HTTP/1.1" 200 - I can see the python listening on port 5000 (mybe this is my problem) netstat -tulpn | grep 5000 tcp6 0 0 :::5000 :::* LISTEN 1976/python I cannot find anything about this error and I can't figure out what is solution. Ubuntu 18.04.5 LTS 4.15.0-136-generic I also tried with 4.20.0-042000-generic but the erorr was the same. jansz0

3 2

15.2.8 mgr keep crashing every few days
by levin ng 01 Mar '21

01 Mar '21

Hi all, I’d recently deployed ceph 15.2.8 with 3(mon,mgr,rgw,mds) and 4 (osd) total 7 host, however I encountered mgr crash a few times a week, the crashing mgr can be any one of 3. I couldn’t identify the problem behind and here is the crash info, appreciate anyone if you have suggestions that I could narrow it down. Thank you very much. { "assert_condition": "ret == 0", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc", "assert_func": "void Thread::create(const char*, size_t)", "assert_line": 157, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f833addc700 time 2021-02-10T20:00:32.980508+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc: 157: FAILED ceph_assert(ret == 0)\n", "assert_thread_name": "mgr-fin", "backtrace": [ "(()+0x12b20) [0x7f835a51cb20]", "(gsignal()+0x10f) [0x7f8358f6d7ff]", "(abort()+0x127) [0x7f8358f57c35]", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f835c07b735]", "(()+0x27a8fe) [0x7f835c07b8fe]", "(()+0x34cef6) [0x7f835c14def6]", "(DispatchQueue::start()+0x3a) [0x7f835c29697a]", "(AsyncMessenger::ready()+0xcd) [0x7f835c3340cd]", "(Messenger::add_dispatcher_head(Dispatcher*)+0x68) [0x7f835c3f8478]", "(MonClient::get_monmap_and_config()+0xbb) [0x7f835c3f66ab]", "(ceph_mount_info::init()+0x4d) [0x7f834298435d]", "(()+0x3680f) [0x7f8342cd280f]", "(()+0x19d421) [0x7f835ba5c421]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x1221d4) [0x7f835b9e11d4]", "(()+0x122c55) [0x7f835b9e1c55]", "(()+0x19cf27) [0x7f835ba5bf27]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(_PyFunction_FastCallDict()+0x122) [0x7f835b9b9ec2]", "(_PyObject_FastCallDict()+0x70e) [0x7f835b9bac9e]", "(()+0x10dc70) [0x7f835b9ccc70]", "(_PyObject_FastCallDict()+0x6ec) [0x7f835b9bac7c]", "(PyObject_CallFunctionObjArgs()+0xe8) [0x7f835b9dbd48]", "(_PyEval_EvalFrameDefault()+0x2588) [0x7f835ba5eef8]", "(()+0xf99b4) [0x7f835b9b89b4]", "(()+0x179e60) [0x7f835ba38e60]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x10d5) [0x7f835ba5da45]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0xfa326) [0x7f835b9b9326]", "(()+0x179e60) [0x7f835ba38e60]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(_PyFunction_FastCallDict()+0x122) [0x7f835b9b9ec2]", "(_PyObject_FastCallDict()+0x70e) [0x7f835b9bac9e]", "(()+0x10dc70) [0x7f835b9ccc70]", "(PyObject_Call()+0x4b) [0x7f835b9c1acb]", "(PyObject_CallMethod()+0x10b) [0x7f835ba5ac6b]", "(ActivePyModule::handle_command(ModuleCommand const&, MgrSession const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)+0x222) [0x55bc0b8a0cb2]", "(()+0x1b0fdd) [0x55bc0b8f5fdd]", "(Context::complete(int)+0xd) [0x55bc0b8b0bdd]", "(Finisher::finisher_thread_entry()+0x1a5) [0x7f835c10b465]", "(()+0x814a) [0x7f835a51214a]", "(clone()+0x43) [0x7f8359032f23]" ], "ceph_version": "15.2.8", "crash_id": "2021-02-10T20:00:32.989661Z_201fd5fb-6e0a-4b50-8a95-fdf9ed9aeb81", "entity_name": "mgr.sds01-cp.cwcxek", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "e1c15d685283e7598b128a37a328ba86ec433dfef97597ac9453b5d52608feda", "timestamp": "2021-02-10T20:00:32.989661Z", "utsname_hostname": "sds01-cp", "utsname_machine": "x86_64", "utsname_release": "4.18.0-240.10.1.el8_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Dec 16 03:30:52 EST 2020" }

4 5

'ceph df' %USED explanation
by Mark Johnson 01 Mar '21

01 Mar '21

I'm in the middle of increasing PG count for one of our pools by making small increments, waiting for the process to complete, rinse and repeat. I'm doing it this way so I can control when all this activity is happening and keeping it away from the busier production traffic times. I'm expecting some inbalance as PGs get created on already unbalanced OSDs, however our monitoring picked up something today that I'm not really understanding. Our total utilization is just over 50% and about 96% of our total data is in this one pool. Due to there not being enough PGs, the amount of data in each is quite large and since they aren't evenly spread across the OSDs, there's a bit of inbalance. That's all cool and to be expected, which is the reason for increasing the PG count in the first place. However, as some PGs are splitting, the new PGs are sometimes being created on OSDs that already have a disproportionate amount of data. Again, not totally unexpected. Our monitoring detected the usage of this pool to be >85% today as I neared the end of another increase in PG count. What I'm not understanding is how this value is determined. I've read other posts and the calculations suggested don't give a result that equals what shows in my %USED column. I'm suspecting that it's somehow related to the MAX AVAIL value (which I believe is somewhat indirectly related to the amount available based on the individual OSD utilization), but none of the posts I read mention this in their calculations and I've been unable to create a formula with any of the values I have to end up with the &USED value I have. For the record, my current total utilization based on a 'ceph osd df' looks like this: TOTAL 39507G(SIZE) 19931G(USE) 17568G(AVAIL) 50.45(%USE) My most utilised OSD (currently in the process of moving some data off this OSD) is 81.58% used with 188G available and a variance of 1.62. A cut-down output of 'ceph df' looks like this: GLOBAL: SIZE AVAIL RAW USED %RAW USED 39507G 17569G 19930G 50.45 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS default.rgw.buckets.data 30 9552G 86.05 1548G 36285066 I suspect that as I get the utilization of my over-utilized OSDs down, this %USED value will drop. But, I'd just love to fully understand how this value is calculated. Thanks, Mark J

1 0