- ceph-users - lists.ceph.io

Backups of monitor
by Michal Strnad 12 Feb '21

12 Feb '21

Hi all, We are looking for a proper solution for backups of monitor (all maps that they hold). On the internet we found advice that we have to stop one of monitor, back it up (dump) and start daemon again. But this is not right approach due to risk of loosing quorum and need of synchronization after monitor is back online. Our goal is to have at least some (recent) metadata of objects in cluster for the last resort when all monitors are in very bad shape/state and we could start any of them. Maybe there is another approach but we are not aware of it. We are running the latest nautilus and three monitors on every cluster. Ad. We don't want to use more monitors than thee. Thank you Cheers Michal -- Michal Strnad

3 2

Re: Cannot access "Object Gateway" in dashboard after setting rgw api keys
by Alfonso Martinez Hidalgo 12 Feb '21

12 Feb '21

Hi Troels, 1) It seems you need to set up the user id like this: ceph dashboard set-rgw-api-user-id <user_id> More info here: https://docs.ceph.com/en/nautilus/mgr/dashboard/#enabling-the-object-gatewa… 2) Have you set up multisite configuration (realms/zonegroups/zones) ? Please paste the output of: radosgw-admin realm list radosgw-admin zonegroup list radosgw-admin zone list Regards, -- Alfonso Martínez Senior Software Engineer, Ceph Storage Red Hat <https://www.redhat.com> <https://red.ht/sig>

2 1

Network design issues
by Frank Schilder 12 Feb '21

12 Feb '21

Dear cephers, I believe we are facing a bottleneck due to an inappropriate overall network design and would like to hear about experience and recommendations. I start with a description of the urgent problem/question and follow up with more details/questions. These observations are on our HPC home file system served with ceph. It has 12 storage servers facing 550+ client servers. Under high load, I start seeing "slow ping time" warnings with quite incredible latencies. I suspect we have a network bottleneck. On the storage servers we have 6x10G LACP trunks. Clients are on single 10G NICs. We have separate VLANs for front- and back network, but they both go through all NICs in the same way, so, technically, its just one cluster network shared with clients. The aggregated bandwidth is sufficient for a single-node storage server load (roughly matches with disk controller IO capacity). However, point-to-point connections are 10G only and I believe that we start observing clients saturating a 10G link and starving all other ceph cluster traffic that needs to go through this link as well. This, in turn, leads to backlog effects with slow ops on unrelated OSDs, affecting overall user experience. The number of OSDs reporting slow ping times is about the percentage one would expect if one or two 10G links are congested. Its usually just one storage server that coughs up. I guess the users with aggressive workloads getting the full bandwidth are happy, but everyone else is complaining. What I observe is that one or two clients can DOS everyone else. I typically see a very high read bandwidth from a few OSDs only and my suspicion is that this is a large job of 50-100 nodes starting the same application at the same time. For example, 50-100 clients reading the same executable simultaneously. I see 5-6GB/s and up to 10K IOP/s read, which is really good in principle. Except that is not fair-shared with other users. Question: I start considering to enable QOS on the switches for traffic between storage servers and would like to know if anyone is doing this and what the experience is. Unfortunately, our network design is probably flawed and makes this now difficult; see below. More Info. Our FS data pool is EC 8+2. I have fast-read enabled. Hence, the network traffic amplification for both, read and write, is quite substantial. Our network is a spine-leaf architecture where ceph servers and ceph clients are distributed more or less equally over the leaf switches. I'm afraid that this is a first flaw in the design, because storage servers and clients compete for the same switches and the clients greatly outnumber the storage servers. It also makes implementing QOS a real pain while it could be just traffic shaping on an uplink trunk to clients if the storage servers were isolated. This is the first design question: Isolated storage cluster providing service via uplinks/gateways versus "integrated/hyper-converged" where storage servers and clients are distributed equally over a spine-leaf architecture. Pros and cons? We have a 100G spine VLT-pair with ports configured as 40G. Up-links from leafs are 2x40, in fact, we have these leafs configured as VLT-pairs for HA as well. A pair has 2x2x40G uplinks and 2x40G VLT interlinks. There are 2 ceph servers per VLT leaf-pair and ca. 85+ client servers on the same pair. There are also clients on leaf switches without ceph servers. I don't think the 40G uplinks are congested, but you never know. We started with the ceph servers having 15HDDs for fs data and 1 SSD for fs meta-data each. With this configuration, the disk speed was the bottleneck and I observed slow ops under high load, but everything was more or less stable. I recently changed an MDS setting that greatly improved both, client performance and also the client's ability to overload OSDs. In addition, one week ago I added 20HDDs in a JBOD per host, which more than doubled the HDD throughput. Both increases in performance together have now the counter-intuitive effect that aggregated performance has tripled in comparison to 2 months ago, but the user experience is very erratic. My suspicion is, as explained above, that each server can now handle a volume of traffic that easily saturates a 10G link, leading to observations that seem to indicate insufficient network capacity whenever too many client/cluster requests go through the same 10G link. In essence, we increased aggregated performance greatly but users complain more than ever. I suspect that this imbalance of server throughput ability and 10G point-to-point limitation is a problem. However, I cannot change the networking and would like some advice of how similar set-ups are configured and if QOS can help. My idea is to enable dot1p layer 2 QOS and give traffic coming from ports with storage servers connected a higher priority than traffic coming from everywhere else. I know it would be a lot simpler if the storage cluster was isolated, but I have to deal with the situation as is for now. Any advice and experience is highly appreciated. If I do it, should I do QOS on both, front- and back network, or is QOS on the VLAN for back-network enough? Note that MONs are only on the front network. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

CEPHFS - MDS gracefull handover of rank 0
by Martin Hronek 12 Feb '21

12 Feb '21

Hello fellow CEPH-users, currently we are updating our CEPH(14.2.16) and making changes to some config settings. TLDR: is there a way to make a graceful MDS active node shutdown without loosing the caps, open files and client connections? Something like handover active state, promote standby to active, ...? Sadly we run into some difficulties when restarting MDS Nodes. While we had two active nodes and one standby we initially though that this would have a nice handover when restarting the active rank ... sadly we saw how the node was going through the states: replay-reconnect-rejoin-active as nicely visualized here https://docs.ceph.com/en/latest/cephfs/mds-states/ This left some nodes going into timeouts until the standby node has gone into the active state again, most probably since the cephfs hast already some 600k folders and 3M files and from the client side it took more than 30s. So before the next MDS the FS config where changed to one active and one standby-replay node, the idea was that since the MDS replay nodes follows the active one the handover would be smoother. The active state was reached faster, but we still noticed some hiccups on the clients while the new active MDS was waiting for clients to reconnect(state up:reconnect) after the failover. The next idea was to do a manual node promotion, graceful shutdown or something similar - where the open caps and sessions would be handed over ... but I did not find any hint in the docs regarding this functionality. But, this should somehow be possible (imho), since when adding a second active mds node (max_mds 2) and then removing it again (max_mds 1) the rank 1 node goes to stopping-state and hands over all clients/caps to rank 0 without interruptions for the clients. Therefore my question: how can one gracefully shutdown an active rank 0 mds node or promote an standby node to the active state without loosing open files/caps or client sessions? Thanks in advance, M

4 6

cephadm upgrade issue persisting with one node
by Darrin Hodges 11 Feb '21

11 Feb '21

Hi all, Still getting upgrade issue with cephadm " Upgrade: failed to pull target image". On each of the nodes in the cluster I can do: docker pull docker.io/ceph/ceph:v15.2.8 And there is no error but the upgrade command fails still. I can see an entry in the logs for: Feb 11 22:27:56 ceph-admin01 bash[2641]: audit 2021-02-11T22:27:55.339997+0000 mon.ceph-admin01 (mon.0) 61 : audit [INF] from='mgr.12064184 ' entity='mgr.ceph-osd01.rpgexq' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/upgrade_state","val":"{\"target_name\": \"docker.io/ceph/ceph:v15.2.8\", \"progress_id\": \"7cf2e315-6cfe-4e9a-88bc-ec8d611b6b4f\", \"error\": \"UPGRADE_FAILED_PULL: Upgrade: failed to pull target image\", \"paused\": true}"}]: dispatch any ideas how I can find extra info on what is going on there? thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

1 1

ceph osd df results
by Marc 11 Feb '21

11 Feb '21

Should the ceph osd df results not have this result for every device class? I do not think that there people mixing these classes in pools. MIN/MAX VAR: 0.78/4.28 STDDEV: 6.15

2 1

MultiSite Sync Problem and Shard Number Relation?
by George Yil 11 Feb '21

11 Feb '21

Hi, I am sort of newby to RGW multisite. I guess there is an important limitation about bucket index sharding if you run multisite. I would like to learn better or correct myself. And I also want to leave a bookmark here for future cephers if possible. I apologize if this is asked before however I am not able to find a good explanation from the mailing list. AFAIK RGW needs bucket indexes as to serve bucket listing. This is optional but mostly demanded. Now you have to set a value for the number of shards of the bucket. This is not dynamic since multisite does not support dynamic resharding yet. AFAIK if your bucket hits the shard limit then your bucket sync is in trouble or maybe stopped. And you should plan a cluster outage in order to reshard the bucket and resync from start. I would like to verify if this is a correct assumption? Here is the problem I've encounter right now. I have 2 nautilus 14.2.9 RGW multisite clusters. Since I was not fully aware of multisite/bucket limitations, I find myself with a fairly large bucket with 256 millions of objects. Now my problem is that bucket syncing seems to be stopped or stalled. I am %100 certain of which but I failed to figure out the exact problem from "radosgw-admin bucket sync status". It is quite possible that I maybe still don't know the proper tool to figure out the root cause. From the RGW logs I could capture this: "check_bucket_shards: resharding needed: stats.num_objects=256166901 shard max_objects=75000000". Cluster ceph.conf includes "rgw override bucket index max shards = 750". I suppose I need to reshard this bucket to continue syncing. And the only way to reshard is that I have to stop all RGW services of both clusters, delete the secondary zone, reshard the bucket and resync the bucket from the beginning. Or for a better solution I should divide this large bucket into smaller buckets. This might have no easy way but to migrate with some kind of S3 sync tool (rather a fast one!).

1 1

Proper solution of slow_ops
by Michal Strnad 11 Feb '21

11 Feb '21

Hi all, we are looking for a proper solution of slow_ops. When the disk failed, node is restated ... a lot of slow operations appear. Even if disk (OSD) or node is back again most of slow_ops are still there. On the internet we found only advice that we have to restart monitor. But this is not right approach. Do you have some better solution? How did you treat slow_ops in your production clusters? We are running the latest nautilus on all clusters. Thank you Cheers Michal

3 3

db_devices doesn't show up in exported osd service spec
by Tony Liu 11 Feb '21

11 Feb '21

Hi, When build cluster Octopus 15.2.5 initially, here is the OSD service spec file applied. ``` service_type: osd service_id: osd-spec placement: host_pattern: ceph-osd-[1-3] data_devices: rotational: 1 db_devices: rotational: 0 ``` After applying it, all HDDs were added and DB of each hdd is created on SSD. Here is the export of OSD service spec. ``` # ceph orch ls --service_name osd.osd-spec --export service_type: osd service_id: osd-spec service_name: osd.osd-spec placement: host_pattern: ceph-osd-[1-3] spec: data_devices: rotational: 1 filter_logic: AND objectstore: bluestore ``` Why db_devices doesn't show up there? When I replace a disk recently, when the new disk was installed and zapped, OSD was automatically re-created, but DB was created on HDD, not SSD. I assume this is because of that missing db_devices? I tried to update service spec, the same result, db_devices doesn't show up when export it. Is this some known issue or something I am missing? Thanks! Tony

5 18

RGW/Swift 404 error when listing/deleting a newly created empty bucket
by Mike Cave 11 Feb '21

11 Feb '21

So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation. The behaviour is intermittent. If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now. To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors. We are running Ceph 14.2.16 on all parts of the cluster. I am using the python-swift client for the connection on a CentOS7 machine. Can replicate the results from the mons or an external client as well. I’m willing to share my test script as well if you would like to see how I’m generating the error. Here is a piece of the logs in case I missed something in the interpretation (log level at 20): 14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 ===== 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 14:23:17.069 7faba00df700 10 host=<NameRemoved> 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 14:23:17.069 7faba00df700 -1 res_query() failed 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift 14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry) 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target 14:23:17.069 7faba00df700 10 Starting retarget 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc 14:23:17.069 7faba00df700 5 Found permission: 15 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ====== -- Mike Cave I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day.

1 0