- ceph-users - lists.ceph.io

Hardware feedback before purchasing for a PoC
by Ignacio Ocampo 09 Mar '20

09 Mar '20

Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase: The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*. System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node) Some notes about the configuration: - 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB* As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase 1. Does the hardware components make sense for the *16TB* growth projection? 2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support! -- Ignacio Ocampo

4 5

A fast tool to export/copy a pool
by Vladimir Brik 09 Mar '20

09 Mar '20

I am wondering if there exists a tool, faster than "rados export", that can copy and restore read-only pools (to/from another pool or file system). It looks like "rados export" is very slow because it is single threaded (the best I can tell, --workers doesn't make a difference). Vlad

1 0

ceph df hangs
by Rebecca CH 09 Mar '20

09 Mar '20

Hello When I run 'ceph df' on one of osd nodes, the command get hanged and no output to the screen. How to resolve it? Thanks.

2 1

Re: ceph rbd volumes/images IO details
by M Ranga Swami Reddy 09 Mar '20

09 Mar '20

On Fri, Mar 6, 2020 at 1:06 AM M Ranga Swami Reddy <swamireddy(a)gmail.com> wrote: > Hello, > Can we get the IOPs of any rbd image/volume? > > For ex: I have created volumes via OpenStack Cinder. Want to know > the IOPs of these volumes. > > In general - we can get pool stats, but not seen the per volumes stats. > > Any hint here? Appreciated. >

3 3

Ceph (version 14.2.7) RGW STS AccessDenied
by 曹海旺 09 Mar '20

09 Mar '20

Hi ,I want to use the sts to get a temporary credentials,but I got 403 . The user info : { "user_id": "sr", "display_name": "shiran", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "sr", "access_key": "FBGYU7CUBUACP2OYBE76", "secret_key": "RoovZBBNdlOvVS4LzuuuCsZ1QPA6W81mbNXvonYD" } ], "swift_keys": [], "caps": [ { "type": "roles", "perm": "*" }, { "type": "user-policy", "perm": "*" } ], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } and the role info : { "id": "c983c0df-a49c-4e36-957d-7d709f65e292", "name": "S3Access", "path": "/application_abc/component_xyz/", "arn": "arn:aws:iam:::role/application_abc/component_xyz/S3Access", "create_date": "2020-03-04T08:19:41.24Z", "max_session_duration": 3600, "assume_role_policy_document": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/sr\"]},\"Action\":[\"sts:AssumeRole\"]}]}" }, I use AWA java sdk to AssumeRole ,code like: public static final String ACCESSKEY_STRING = “xxxxxxxxxxxxxx"; public static final String BUCKETNAME_STRING = "osstest"; public static final String SECRETKEY_STRING = “xxxxxxxxxxxxxxxxxxxxx"; public static final String ENDPOINT_STRING = "http://192.168.3.250”;// haproxy String clientRegion = ENDPOINT_STRING; String roleARN = "arn:aws:iam:::role/application_abc/component_xyz/S3Access"; String roleSessionName = "bob"; String bucketName = ENDPOINT_STRING; AWSCredentials credentials = new BasicAWSCredentials(ACCESSKEY_STRING, SECRETKEY_STRING); ClientConfiguration clientConfig = new ClientConfiguration(); clientConfig.setProtocol(Protocol.HTTP); clientConfig.setSignerOverride("AWS3SignerType"); AWSSecurityTokenService sts_client = AWSSecurityTokenServiceClientBuilder.standard() .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(ENDPOINT_STRING, "")) .withCredentials(new AWSStaticCredentialsProvider(credentials)).withClientConfiguration(clientConfig) .build(); GetSessionTokenRequest session_token_request = new GetSessionTokenRequest(); AssumeRoleRequest roleRequest = new AssumeRoleRequest() .withRoleArn(roleARN) .withRoleSessionName(roleSessionName).withTokenCode("sts"); session_token_request.setDurationSeconds(3600); AssumeRoleResult roleResponse = sts_client.assumeRole(roleRequest); Credentials sessionCredentials = roleResponse.getCredentials(); I got the response: <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx000000000000000000016-005e64fa9e-b5b47-default</RequestId><HostId>b5b47-default-default</HostId></Error>" The debug info in the log file like this : 1 ====== starting new request req=0x5592abefc910 ===== 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s initializing for trans_id = tx000000000000000000016-005e64fa9e-b5b47-default 2020-03-08 22:01:02.984 7f2548f95700 10 rgw api priority: s3=7 s3website=6 2020-03-08 22:01:02.984 7f2548f95700 10 host=rgw1 2020-03-08 22:01:02.984 7f2548f95700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2020-03-08 22:01:02.984 7f2548f95700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ 2020-03-08 22:01:02.984 7f2548f95700 10 meta>> HTTP_X_AMZ_DATE 2020-03-08 22:01:02.984 7f2548f95700 10 meta>> HTTP_X_AMZN_AUTHORIZATION 2020-03-08 22:01:02.984 7f2548f95700 10 x>> x-amz-date:Sun, 08 Mar 2020 14:01:02 GMT 2020-03-08 22:01:02.984 7f2548f95700 10 x>> x-amzn-authorization:AWS3 AWSAccessKeyId=FBGYU7CUBUACP2OYBE76,Algorithm=HmacSHA256,SignedHeaders=Host;X-Amz-Date,Signature=D7p74AFsJn4J823e6Up4Zo2eRV/n5PFmyQ2/w+kOaqY= 2020-03-08 22:01:02.984 7f2548f95700 20 get_handler handler=26RGWHandler_REST_Service_S3 2020-03-08 22:01:02.984 7f2548f95700 10 handler=26RGWHandler_REST_Service_S3 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s getting op 4 2020-03-08 22:01:02.984 7f2548f95700 10 Content of POST: Action=AssumeRole&Version=2011-06-15&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2Fapplication_abc%2Fcomponent_xyz%2FS3Access&RoleSessionName=bob&SerialNumber=sdfsdf&TokenCode=sts 2020-03-08 22:01:02.984 7f2548f95700 10 Key: ActionValue: AssumeRole 2020-03-08 22:01:02.984 7f2548f95700 10 Key: VersionValue: 2011-06-15 2020-03-08 22:01:02.984 7f2548f95700 10 Key: RoleArnValue: arn:aws:iam:::role/application_abc/component_xyz/S3Access 2020-03-08 22:01:02.984 7f2548f95700 10 Key: RoleSessionNameValue: bob 2020-03-08 22:01:02.984 7f2548f95700 10 Key: SerialNumberValue: sdfsdf 2020-03-08 22:01:02.984 7f2548f95700 10 Key: TokenCodeValue: sts 2020-03-08 22:01:02.984 7f2548f95700 10 req 22 0.000s sts:assume_role scheduling with dmclock client=0 cost=1 2020-03-08 22:01:02.984 7f2548f95700 10 op=16RGWSTSAssumeRole 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s sts:assume_role verifying requester 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::s3::S3AnonymousEngine granted access 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::s3::AWSAuthStrategy granted access 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s sts:assume_role normalizing buckets and tenants 2020-03-08 22:01:02.984 7f2548f95700 10 s->object=<NULL> s->bucket= 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s sts:assume_role init permissions 2020-03-08 22:01:02.985 7f2548f95700 20 get_system_obj_state: rctx=0x5592abefb400 obj=default.rgw.meta:users.uid:anonymous state=0x5592aa8d99c0 s->prefetch_data=0 2020-03-08 22:01:02.985 7f2548f95700 10 cache get: name=default.rgw.meta+users.uid+anonymous : type miss (requested=0x6, cached=0x0) 2020-03-08 22:01:02.985 7f2548f95700 20 WARNING: blocking librados call 2020-03-08 22:01:02.985 7f2548f95700 1 -- 192.168.3.247:0/1491423941 --> [v2:192.168.3.244:6800/2853,v1:192.168.3.244:6801/2853] -- osd_op(unknown.0.0:159377 3.0 3:0ab27964:users.uid::anonymous:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e282) v8 -- 0x5592abea3b80 con 0x5592ab6acc00 2020-03-08 22:01:02.985 7f2593ab4700 1 -- 192.168.3.247:0/1491423941 <== osd.5 v2:192.168.3.244:6800/2853 4 ==== osd_op_reply(159377 anonymous [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 195+0+0 (crc 0 0 0) 0x5592abda1400 con 0x5592ab6acc00 2020-03-08 22:01:02.985 7f2548f95700 10 cache put: name=default.rgw.meta+users.uid+anonymous info.flags=0x0 2020-03-08 22:01:02.985 7f2548f95700 10 moving default.rgw.meta+users.uid+anonymous to cache LRU end 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role recalculating target 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role reading permissions 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role init op 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role verifying op mask 2020-03-08 22:01:02.985 7f2548f95700 20 req 22 0.001s sts:assume_role required_mask= 0 user.op_mask=7 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role verifying op permissions 2020-03-08 22:01:02.985 7f2548f95700 20 get_system_obj_state: rctx=0x5592abefb7f0 obj=default.rgw.meta:roles:role_names.S3Access state=0x5592aa8d99c0 s->prefetch_data=0 2020-03-08 22:01:02.985 7f2548f95700 10 cache get: name=default.rgw.meta+roles+role_names.S3Access : type miss (requested=0x6, cached=0x3) 2020-03-08 22:01:02.985 7f2548f95700 20 WARNING: blocking librados call 2020-03-08 22:01:02.985 7f2548f95700 1 -- 192.168.3.247:0/1491423941 --> [v2:192.168.3.245:6808/2863,v1:192.168.3.245:6809/2863] -- osd_op(unknown.0.0:159378 3.3 3:c86e03bb:roles::role_names.S3Access:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e282) v8 -- 0x5592abf18000 con 0x5592ab6ad400 2020-03-08 22:01:02.986 7f2593ab4700 1 -- 192.168.3.247:0/1491423941 <== osd.8 v2:192.168.3.245:6808/2863 3 ==== osd_op_reply(159378 role_names.S3Access [getxattrs,stat] v0'0 uv4 ondisk = 0) v8 ==== 205+0+20 (crc 0 0 0) 0x5592abda1400 con 0x5592ab6ad400 2020-03-08 22:01:02.986 7f2548f95700 10 cache put: name=default.rgw.meta+roles+role_names.S3Access info.flags=0x6 2020-03-08 22:01:02.986 7f2548f95700 10 moving default.rgw.meta+roles+role_names.S3Access to cache LRU end 2020-03-08 22:01:02.986 7f2548f95700 20 get_system_obj_state: s->obj_tag was set empty 2020-03-08 22:01:02.986 7f2548f95700 10 cache get: name=default.rgw.meta+roles+role_names.S3Access : hit (requested=0x1, cached=0x7) 2020-03-08 22:01:02.986 7f2548f95700 20 get_system_obj_state: rctx=0x5592abefb7f0 obj=default.rgw.meta:roles:roles.c983c0df-a49c-4e36-957d-7d709f65e292 state=0x5592aa8d99c0 s->prefetch_data=0 2020-03-08 22:01:02.986 7f2548f95700 10 cache get: name=default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 : type miss (requested=0x6, cached=0x3) 2020-03-08 22:01:02.986 7f2548f95700 20 WARNING: blocking librados call 2020-03-08 22:01:02.986 7f2548f95700 1 -- 192.168.3.247:0/1491423941 --> [v2:192.168.3.244:6800/2853,v1:192.168.3.244:6801/2853] -- osd_op(unknown.0.0:159379 3.0 3:0e42bcf0:roles::roles.c983c0df-a49c-4e36-957d-7d709f65e292:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e282) v8 -- 0x5592abf182c0 con 0x5592ab6acc00 2020-03-08 22:01:02.986 7f2593ab4700 1 -- 192.168.3.247:0/1491423941 <== osd.5 v2:192.168.3.244:6800/2853 5 ==== osd_op_reply(159379 roles.c983c0df-a49c-4e36-957d-7d709f65e292 [getxattrs,stat] v0'0 uv4 ondisk = 0) v8 ==== 228+0+20 (crc 0 0 0) 0x5592abda1400 con 0x5592ab6acc00 2020-03-08 22:01:02.986 7f2548f95700 10 cache put: name=default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 info.flags=0x6 2020-03-08 22:01:02.986 7f2548f95700 10 moving default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 to cache LRU end 2020-03-08 22:01:02.986 7f2548f95700 20 get_system_obj_state: s->obj_tag was set empty 2020-03-08 22:01:02.986 7f2548f95700 10 cache get: name=default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 : hit (requested=0x1, cached=0x7) 2020-03-08 22:01:02.986 7f2548f95700 20 op->ERRORHANDLER: err_no=-1 new_err_no=-1 2020-03-08 22:01:02.986 7f2548f95700 2 req 22 0.002s sts:assume_role op status=0 2020-03-08 22:01:02.986 7f2548f95700 2 req 22 0.002s sts:assume_role http status=403

2 1

RGW jaegerTracing
by Abhinav Singh 09 Mar '20

09 Mar '20

I am trying to implement jaeger tracing in RGW, I need some advice regarding on which functions should I actually tracing to get a good actual performance status of clusters Till now I am able to deduce followings : 1.I think we need to provide tracing functions where the `rgw` is communicating with the librados, (particularly the librgw where the communication is actually happening), because http request and response not to be considered for tracing because that depends on clients internet speed. 2.In librgw the functions like this here <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> and its corresponding overloading methods and also the this function here <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> and its corresponding overloaded functions. 3.I see that pools are ultimately used to enter the crush algorithm for writing data, so I think the ceation of pools should also be taken into account while tracing,(creation of pool should be main span and these functions <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> should be its child span). Functionality of buckets like that of this <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> do not require tracing beacuse they are http requests. Any kind of guidance will be of great help. Thank You.

2 1

log_latency_fn slow operation
by 徐蕴 08 Mar '20

08 Mar '20

Hi, Our cluster (14.2.6) has sporadic slow ops warnings since upgrading from Jewel 1 month ago. Today I checked the OSD log files and found out a lot of entries like: ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.16871s, txc = 0x55e33ae41b80 ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.15158s, txc = 0x55e3639b3340 ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.77361s, txc = 0x55e3379cc840 ceph-osd.5.log:2020-03-04 10:33:52.666 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.42519s, txc = 0x55e33722d600 or /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 00:41:31.110 7f3dc0bc8700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for submit_transact, latency = 8.1279s /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 00:41:31.110 7f3dd1bea700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for kv_final, latency = 7.88786s /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 02:21:35.180 7f3dd1bea700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for kv_final, latency = 6.06171s /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 05:31:30.298 7f3dc1bca700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for submit_transact, latency = 5.34228s The cluster setup is: SATA SSD (as DB) + SATA HDD 1:3. Any suggest how to debug this problem? Thank you! br, Xu Yun

2 1

Welcome to the "ceph-users" mailing list
by Abhinav Singh 08 Mar '20

08 Mar '20

singhabhinav9051571833(a)gmail.com

1 0

Disabling Telemetry
by m＠silvenga.com 07 Mar '20

07 Mar '20

Is there another way to disable telemetry then using: > ceph telemetry off > Error EIO: Module 'telemetry' has experienced an error and cannot handle commands: cannot concatenate 'str' and 'UUID' objects I'm attempting to get all my clusters out of a constant HEALTH_ERR state caused by either the above error or the telemetry endpoint being down.

2 2

High CPU usage by ceph-mgr in 14.2.6
by jbardgett＠godaddy.com 07 Mar '20

07 Mar '20

After upgrading one of our clusters from Luminous 12.2.12 to Nautilus 14.2.6, I am seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). The way we found this was due to Prometheus being unable to report out certain pieces of data, specifically OSD Usage, OSD Apply and Commit Latency. Which are all similar issues people were having in previous versions of Nautilus. Bryan Stillwell reported this previously on a separate cluster, 14.2.5, we have here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VW3GNVJGOOW… That issue was resolved with the upgrade to 14.2.6. We are seeing a similar issue on this other cluster with a couple differences. This cluster has 1900+ OSD in it, the previous one had 300+ The top user is libceph-common, instead of mmap 4.86% libceph-common.so.0 [.] EventCenter::create_time_event 2.78% [kernel] [k] nmi 2.64% libstdc++.so.6.0.19 [.] __dynamic_cast On all our other clusters that have been upgraded to 14.2.6 we are not experiencing this issue, the next largest being 800+ OSD. We feel this is related to the size of the cluster, similarly to the previous report. Anyone else experiencing this and/or can provide some direction on how to go about resolving this? Thanks, Joe

6 7