- ceph-users - lists.ceph.io

Telemetry ident use?
by Matthew Vernon 17 Mar '21

17 Mar '21

Hi, What use is made of the ident data in the telemetry module? It's disabled by default, and the docs don't seem to say what it's used for... Thanks, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

1 0

Erasure-coded Block Device Image Creation With qemu-img - Help
by duluxoz 17 Mar '21

17 Mar '21

Hi Guys, So, new issue (I'm gonna get the hang of this if it kills me :-) ). I have a working/healthy Ceph (Octopus) Cluster (with qemu-img, libvert, etc, installed), and an erasure-coded pool called "my_pool". I now need to create a "my_data" image within the "my_pool" pool. As this is for a KVM host / block device (hence the qemu-img et.al.) I'm attempting to use qemu-img, so the command I am using is: ``` qemu-img create -f rbd rbd:my_pool/my_data 1T ``` The error message I received was: ``` qemu-img: rbd:my_pool/my_data: error rbd create: Operation not supported ``` So, I tried the 'raw' rbd command: ``` rbd create -s 1T my_pool/my_data ``` and got the error: ``` _add_image_to_directory: error adding image to directory: (95) Operation not supported rbd: create error: (95) Operation not supported ``` So I don't believe the issue is with the 'qemu-img' command - but I may be wrong. After doing some research I *think* I need to specify a replicated (as opposed to erasure-coded) pool for my_pool's metadata (eg 'my_pool_metadata'), and thus use the command: ``` rbd create -s 1T --data-pool my_pool my_pool_metadata/my_data ``` First Question: Is this correct? Second Question: What is the qemu-img equivalent command - is it: ``` qemu-img create -f rbd rbd:--data-pool my_pool my_pool_metadata/my_data 1T ``` or something similar? Thanks in advance Dulux-Oz

3 2

Quick quota question
by Andrew Walker-Brown 17 Mar '21

17 Mar '21

Hi all When setting a quota on a pool (or directory in Cephfs), is it the amount of client data written or the client data x number of replicas that counts toward the quota? Cheers A Sent from my iPhone

4 6

Diskless boot for Ceph nodes
by Stephen Smith6 17 Mar '21

17 Mar '21

6 6

Networking Idea/Question
by Dave Hall 17 Mar '21

17 Mar '21

Hello, If anybody out there has tried this or thought about it, I'd like to know... I've been thinking about ways to squeeze as much performance as possible from the NICs on a Ceph OSD node. The nodes in our cluster (6 x OSD, 3 x MGR/MON/MDS/RGW) currently have 2 x 10GB ports. Currently, one port is assigned to the front-side network, and one to the back-side network. However, there are times when the traffic on one side or the other is more intense and might benefit from a bit more bandwidth. The idea I had was to bond the two ports together, and to run the back-side network in a tagged VLAN on the combined 20GB LACP port. In order to keep the balance and prevent starvation from either side it would be necessary to apply some sort of a weighted fair queuing mechanism via the 'tc' command. The idea is that if the client side isn't using up the full 10GB/node, and there is a burst of re-balancing activity, the bandwidth consumed by the back-side traffic could swell to 15GB or more. Or vice versa. From what I have read and studied, these algorithms are fairly responsive to changes in load and would thus adjust rapidly if the demand from either side suddenly changed. Maybe this is a crazy idea, or maybe it's really cool. Your thoughts? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

7 11

Ceph Cluster Taking An Awful Long Time To Rebalance
by duluxoz 17 Mar '21

17 Mar '21

Hi Guys, Is the below "ceph -s" normal? This is a brand new cluster with (at the moment) a single Monitor and 7 OSDs (each 6 GiB) that has no data in it (yet), and yet its taking almost a day to "heal itself" from adding in the 2nd OSD. ~~~ cluster: id: [REDACTED] health: HEALTH_WARN Reduced data availability: 256 pgs inactive, 256 pgs incomplete Degraded data redundancy: 12 pgs undersized services: mon: 1 daemons, quorum [REDACTED] (age 22h) mgr: [REDACTED](active, since 22h) osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs data: pools: 5 pools, 288 pgs objects: 7 objects, 0 B usage: 7.1 GiB used, 38 TiB / 38 TiB avail pgs: 88.889% pgs not active 6/21 objects misplaced (28.571%) 256 creating+incomplete 18 active+clean 12 active+undersized+remapped 2 active+clean+remapped progress: Rebalancing after osd.1 marked in (22h) [............................] PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h) [............................] ~~~ Thanks in advance Matthew J -- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.

3 6

Has anyone contact Data for Samsung Datacenter SSD Support ?
by Christoph Adomeit 16 Mar '21

16 Mar '21

Hi, I hope someone here can help me out with some contact data, email-adress or phone Number for Samsung Datacenter SSD Support ? If I contact Standard Samsung Datacenter Support they tell me they are not there to support PM1735 Drives. We are planning a new Ceph-Cluster and we are thinking of Samsung PM1735 NVME u.2 ssds Unfortunately the PM1735 is not available with u2 interface but the PM1733 is. Some manager from Samsung once told me that PM1733 and PM1735 is exactly the same Hardware, it is only provisioned differently. But he did not know whom to ask. Any idea of whom I could contact at Samsung or of how to provision the PM1733 (7.6TB) to a PM1735 (6.4TB) . I want the provisioning for better DWPD (3DWPD instead of 1DWPD).

3 2

RGW dashboard
by thomas.charles＠virttom.io 16 Mar '21

16 Mar '21

HI, this is regarding my fresh installation of CEPH on a Proxmox cluster. I need to use the radosGW and I'm quite stucked in dashboard configuration. ceph -v ceph version 15.2.9 (7b3df4a1b15c5a048c237733c797a2667f08196e) octopus (stable) ceph -w cluster: id: 86c332e8-9e83-4a11-bd10-7fc34d4fbd0a health: HEALTH_OK services: mon: 3 daemons, quorum srv157ilo152,srv158ilo151,srv161ilo150 (age 8d) mgr: srv157ilo152(active, since 24h), standbys: srv161ilo150, srv158ilo151 osd: 15 osds: 15 up (since 8d), 15 in (since 3w) rgw: 3 daemons active (radosgw.srv157ilo152, radosgw.srv158ilo151, radosgw.srv161ilo150) everything seems correct following the (excellent) documentation but i miss something on RGW dashboard configuration : radosgw-admin user create --uid=111 --display-name=cephrgwdash --system gave me an access_key and a secret_key. So I created 2 files rgw_acces and rgw_secret on my /etc/pve/priv folder and paste keys on it next is the error : ceph dashboard set-rgw-api-access-key -i rgw-acces Invalid command: missing required parameter value(<string>) dashboard set-rgw-api-access-key <value> : Set the RGW_API_ACCESS_KEY option value Error EINVAL: invalid command sorry if I made beginner mistake, I'm not used to CEPH and more than that I'm not the devops guy but the hardware / virtualization one tryin to improve its skills. ;) regards, Virttom007

1 0

osd_max_backfills = 1 for one OSD
by Dave Hall 16 Mar '21

16 Mar '21

Hello, I've been trying to get an OSD ready for removal for about a week. RIght now I have 2 pgs backfilling and 21 in backfill_wait. Looking around I found reference to the following command (followed by my output). # ceph config dump | egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" global advanced osd_max_backfills 10 osd advanced osd_max_backfills 20 osd advanced osd_recovery_max_active 20 osd.1 advanced osd_max_backfills 1 I'm not sure why osd.1 is special. I'd like to know how to get rid of this. If I increase this value for osd.1 it will likely speed up my backfills, but the special setting for osd.1 will persist. There must be a way to unset this so the general setting is applied, right? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

Inactive pg, how to make it active / or delete
by Szabo, Istvan (Agoda) 16 Mar '21

16 Mar '21

Hi, I have 4 inactive pg in my cluster, the osds are dies where it was before. How can I make it work again? Maybe just threw away because last backfill=max? Based on the pg query it is totally up on other osds. It is an EC 3+1. This is an example inactive pg: ceph pg 44.1f0 query { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "incomplete", "epoch": 5541839, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "info": { "pgid": "44.1f0s0", "last_update": "4863820'2109288", "last_complete": "4863820'2109288", "log_tail": "3881944'2103139", "last_user_version": 11189093, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "4863820'2109288", "reported_seq": "12355046", "reported_epoch": "5541839", "state": "incomplete", "last_fresh": "2021-03-16T10:39:21.058569+0100", "last_change": "2021-03-16T10:39:21.058569+0100", "last_active": "2021-01-20T01:07:18.246158+0100", "last_peered": "2021-01-20T01:07:13.931842+0100", "last_clean": "2021-01-20T01:07:07.392736+0100", "last_became_active": "2021-01-20T01:07:09.187047+0100", "last_became_peered": "2021-01-20T01:07:09.187047+0100", "last_unstale": "2021-03-16T10:39:21.058569+0100", "last_undegraded": "2021-03-16T10:39:21.058569+0100", "last_fullsized": "2021-03-16T10:39:21.058569+0100", "mapping_epoch": 5541572, "log_start": "3881944'2103139", "ondisk_log_start": "3881944'2103139", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 9, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 6149, "ondisk_log_size": 6149, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 356705195545, "num_objects": 98594, "num_object_clones": 758, "num_object_copies": 394376, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 98594, "num_whiteouts": 750, "num_read": 30767, "num_read_kb": 37109881, "num_write": 31023, "num_write_kb": 66655410, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 1000136, "num_bytes_recovered": 3631523354253, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 68 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 4865266, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "34(1)", "pgid": "44.1f0s1", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "119", "reported_epoch": "5520802", "state": "down", "last_fresh": "2021-03-14T07:03:14.383913+0100", "last_change": "2021-03-14T07:03:14.383913+0100", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "2021-03-14T07:03:14.383913+0100", "last_undegraded": "2021-03-14T07:03:14.383913+0100", "last_fullsized": "2021-03-14T07:03:14.383913+0100", "mapping_epoch": 5541572, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 46, 68 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 1, "dne": 0, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "54(3)", "pgid": "44.1f0s3", "last_update": "4863820'2109288", "last_complete": "4863820'2109288", "log_tail": "3881944'2103139", "last_user_version": 11189093, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "4863820'2109288", "reported_seq": "11820744", "reported_epoch": "5336744", "state": "down", "last_fresh": "2021-03-05T16:47:09.479395+0100", "last_change": "2021-03-05T16:45:14.475314+0100", "last_active": "2021-02-15T11:57:07.430181+0100", "last_peered": "2021-01-19T23:23:39.318842+0100", "last_clean": "2021-01-19T23:23:39.318842+0100", "last_became_active": "2021-01-18T23:18:46.888782+0100", "last_became_peered": "2021-01-18T23:18:46.888782+0100", "last_unstale": "2021-03-05T16:47:09.479395+0100", "last_undegraded": "2021-03-05T16:47:09.479395+0100", "last_fullsized": "2021-03-05T16:47:09.479395+0100", "mapping_epoch": 5541572, "log_start": "3881944'2103139", "ondisk_log_start": "3881944'2103139", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 9, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 6149, "ondisk_log_size": 6149, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 356705195545, "num_objects": 98594, "num_object_clones": 758, "num_object_copies": 394376, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 98594, "num_whiteouts": 750, "num_read": 30767, "num_read_kb": 37109881, "num_write": 31023, "num_write_kb": 66655410, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 1000136, "num_bytes_recovered": 3631523354253, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 34, 46, 68, 74, 131 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 4865007, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "62(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "72(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 192123, "epoch_pool_created": 175799, "last_epoch_started": 4865266, "last_interval_started": 4865265, "last_epoch_clean": 4865007, "last_interval_clean": 4865006, "last_epoch_split": 192123, "last_epoch_marked_full": 0, "same_up_since": 5541572, "same_interval_since": 5541572, "same_primary_since": 5520915, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "130", "reported_epoch": "5520914", "state": "down", "last_fresh": "2021-03-14T07:09:19.787519+0100", "last_change": "2021-03-14T07:03:21.216689+0100", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "2021-03-14T07:09:19.787519+0100", "last_undegraded": "2021-03-14T07:09:19.787519+0100", "last_fullsized": "2021-03-14T07:09:19.787519+0100", "mapping_epoch": 5541572, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 192123, "last_epoch_clean": 4865007, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "4863820'2109288", "last_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "last_deep_scrub": "4808731'2109261", "last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100", "last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 46, 34, 62, 74 ], "acting": [ 46, 34, 62, 74 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 33, 34, 46, 68 ], "up_primary": 46, "acting_primary": 46, "purged_snaps": [] }, "empty": 1, "dne": 0, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "74(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "74(3)", "pgid": "44.1f0s3", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "131(2)", "pgid": "44.1f0s2", "last_update": "0'0", "last_complete": "0'0", "log_tail": "0'0", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 0, "epoch_pool_created": 0, "last_epoch_started": 0, "last_interval_started": 0, "last_epoch_clean": 0, "last_interval_clean": 0, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 0, "same_interval_since": 0, "same_primary_since": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": "0", "reported_epoch": "0", "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 0, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": -1, "purged_snaps": [] }, "empty": 1, "dne": 1, "incomplete": 0, "last_epoch_started": 0, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Peering/Incomplete", "enter_time": "2021-03-16T10:39:21.058568+0100", "comment": "not enough complete instances of this PG" }, { "name": "Started/Primary/Peering", "enter_time": "2021-03-16T10:39:21.058483+0100", "past_intervals": [ { "first": "4865006", "last": "5541571", "all_participants": [ { "osd": 33, "shard": 1 }, { "osd": 34, "shard": 1 }, { "osd": 46, "shard": 0 }, { "osd": 54, "shard": 3 }, { "osd": 68, "shard": 2 }, { "osd": 72, "shard": 2 }, { "osd": 74, "shard": 2 }, { "osd": 131, "shard": 2 } ], "intervals": [ { "first": "4865265", "last": "4865269", "acting": "33(1),46(0),68(2)" }, { "first": "4865496", "last": "4865498", "acting": "33(1),46(0),54(3)" }, { "first": "5085650", "last": "5085653", "acting": "34(1),54(3),68(2)" }, { "first": "5114555", "last": "5114581", "acting": "34(1),46(0),68(2)" }, { "first": "5132241", "last": "5132272", "acting": "34(1),46(0),54(3)" }, { "first": "5236601", "last": "5236617", "acting": "46(0),54(3),68(2)" }, { "first": "5520781", "last": "5520798", "acting": "34(1),54(3),72(2)" }, { "first": "5520915", "last": "5520919", "acting": "46(0),54(3),72(2)" }, { "first": "5520920", "last": "5541571", "acting": "34(1),46(0),54(3),72(2)" } ] } ], "probing_osds": [ "34(1)", "46(0)", "54(3)", "62(2)", "72(2)", "74(2)", "74(3)", "131(2)" ], "down_osds_we_would_probe": [ 33, 68 ], "peering_blocked_by": [], "peering_blocked_by_detail": [ { "detail": "peering_blocked_by_history_les_bound" } ] }, { "name": "Started", "enter_time": "2021-03-16T10:39:21.058461+0100" } ], "agent_state": {} } Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2