- ceph-users - lists.ceph.io

0B OSDs?
by DHilsbos＠performair.com 25 Oct '19

25 Oct '19

All; We're setting up our second cluster, using version 14.2.4, and we've run into a weird issue: all of our OSDs are created with a size of 0 B. Weights are appropriate for the size of the underlying drives, but ceph -s shows this: cluster: id: <id> health: HEALTH_WARN Reduced data availability: 256 pgs inactive too few PGs per OSD (28 < min 30) services: mon: 3 daemons, quorum s700041,s700042,s700043 (age 4d) mgr: s700041(active, since 3d), standbys: s700042, s700043 osd: 9 osds: 9 up (since 21m), 9 in (since 44m) data: pools: 1 pools, 256 pgs objects: 0 objects, 0 B -->usage: 0 B used, 0 B / 0 B avail<-- (emphasis added) pgs: 100.000% pgs unknown 256 unknown Thoughts? I have ceph-volumne.log, and the log from one of the OSD daemons, though it looks like the auth keys get printed to the ceph-volume.log. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International inc. DHilsbos(a)PerformAir.com www.PerformAir.com

1 0

minimum osd size?
by Frank R 25 Oct '19

25 Oct '19

Hi all, I have 40 nvme drives with about 20G free space each. Would creating a 10GB partition/lvm on each of the nvmes for an rgw index pool be a bad idea? RGW has about about 5 million objects I don't think space will be an issue but I am worried about the 10G size, is it just too small for a bluestore OSD? thx Frank

3 2

Re: Authentication failure at radosgw for presigned urls
by Biswajeet Patra 25 Oct '19

25 Oct '19

++ceph-users On Fri, Sep 20, 2019 at 4:48 PM Biswajeet Patra < biswajeet.patra(a)flipkart.com> wrote: > Hi, > We recently faced an issue with radosgw authentication of presigned urls. > The presigned url generated by client to download object fails at radosgw > with http error 403 i.e SignatureDoesNotMatch. > > The radosgw computes the signature (*v2 signature in this case*) using > the s3 specification > https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html. > In this process, a string_to_sign is created by concatenating selected > elements from requests which is then authenticated using hmac to create the > final signature. If this signature matches with the client signature, the > authentication is successful or else it fails with SignatureDoesNotMatch > error. As part of the StringToSign, the CanonicalizedAmzHeaders should only > include headers that start with "x-amz-" and ignore other headers. But in > the radosgw code, there are other meta prefixes that are checked against > the http request headers and if matched are included in the > CanonicalizedAmzHeaders to compute the final signature. For e.g, if a > request header contains "HTTP_X_ACCOUNT" its selected by radosgw to include > in amz_headers but the same will be ignored by AWS SDK as it does not start > with "x-amz-". This will result in different signature computed by client > and radosgw. > > Code Snippet: rgw_common.cc > struct str_len meta_prefixes[] = { STR_LEN_ENTRY("HTTP_X_AMZ"), > STR_LEN_ENTRY("HTTP_X_GOOG"), > STR_LEN_ENTRY("HTTP_X_DHO"), > STR_LEN_ENTRY("HTTP_X_RGW"), > STR_LEN_ENTRY("HTTP_X_OBJECT"), > STR_LEN_ENTRY("HTTP_X_CONTAINER"), > STR_LEN_ENTRY("HTTP_X_ACCOUNT"), > {NULL, 0} }; > > The method init_meta_info() which matches the above prefixes is called > from RGWREST::preprocess() which is invoked for all s3 requests. It will be > helpful to know as to why these prefixes that are not specified by AWS S3 > comes in the path of s3 authentication. Was it added for swift use-cases > only? If so, then why its included in rgw_common.cc? > As a proposed fix, we can remove the highlighted meta prefixes that are > not specified by aws from the s3 authentication path signature calculation. > Let me know if you have any queries or solutions. > > Regards, > Biswajeet > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 1

Unbalanced data distribution
by Thomas 25 Oct '19

25 Oct '19

Hi, in my 7 OSD node cluster I have the following disks: Node 1 48x 1.6TB Node 2 48x 1.6TB Node 3 48x 1.6TB Node 4 48x 1.6TB Node 5 48x 7.2TB Node 6 48x 7.2TB Node 7 48x 7.2TB The disk sizes are represented in CRUSH map accordingly. For these disks only I created a pool "hdb_backup" with size 3. Based on output of rados df this pool is using 247TB currently. root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR cephfs_data 345 GiB 99092 0 297276 0 0 0 1638 811 MiB 109235 365 GiB 0 B 0 B cephfs_metadata 102 MiB 48 0 144 0 0 0 8 8 KiB 8588 106 MiB 0 B 0 B hdb_backup 247 TiB 64671398 0 194014194 0 0 0 12902005 4.3 TiB 323647757 601 TiB 0 B 0 B hdd 2.4 TiB 635457 0 1270914 0 0 0 13237278 321 GiB 21526953 3.0 TiB 0 B 0 B nvme 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B ssd 251 GiB 64307 0 128614 0 0 0 615475 29 GiB 885085 55 GiB 0 B 0 B total_objects 65470302 total_used 747 TiB total_avail 784 TiB total_space 1.5 PiB In order to rebalance the data in this pool I have configured balance mode upmap: root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Unfortunately the data distribution is not balanced at all on 1.6TB disks, means the range is between 53.37% and 83.04%. root@ld3955:~# ceph osd df | awk '{ print "osd."$1, "size: "$5, "usage: " $17, "reweight: "$4 }' | sort -nk5 osd.ID size: SIZE usage: reweight: REWEIGHT osd.MIN/MAX size: 26.45 usage: reweight: STDDEV: osd.TOTAL size: TiB usage: reweight: 747 osd.265 size: 1.6 usage: 53.37 reweight: 1.00000 osd.248 size: 1.6 usage: 53.41 reweight: 1.00000 osd.111 size: 1.6 usage: 53.43 reweight: 1.00000 osd.161 size: 1.6 usage: 53.46 reweight: 1.00000 osd.85 size: 1.6 usage: 53.46 reweight: 1.00000 osd.241 size: 1.6 usage: 53.49 reweight: 1.00000 osd.238 size: 1.6 usage: 53.51 reweight: 1.00000 osd.259 size: 1.6 usage: 53.56 reweight: 1.00000 osd.88 size: 1.6 usage: 53.57 reweight: 1.00000 osd.204 size: 1.6 usage: 53.58 reweight: 1.00000 osd.159 size: 1.6 usage: 55.16 reweight: 1.00000 osd.81 size: 1.6 usage: 55.16 reweight: 1.00000 osd.116 size: 1.6 usage: 55.20 reweight: 1.00000 osd.195 size: 1.6 usage: 55.25 reweight: 1.00000 osd.169 size: 1.6 usage: 55.33 reweight: 1.00000 osd.158 size: 1.6 usage: 55.34 reweight: 1.00000 [...] osd.146 size: 1.6 usage: 79.31 reweight: 1.00000 osd.140 size: 1.6 usage: 79.34 reweight: 0.89999 osd.262 size: 1.6 usage: 79.38 reweight: 0.89999 osd.217 size: 1.6 usage: 79.48 reweight: 1.00000 osd.83 size: 1.6 usage: 79.50 reweight: 1.00000 osd.239 size: 1.6 usage: 79.52 reweight: 0.79999 osd.190 size: 1.6 usage: 80.87 reweight: 1.00000 osd.97 size: 1.6 usage: 80.95 reweight: 1.00000 osd.216 size: 1.6 usage: 80.97 reweight: 1.00000 osd.160 size: 1.6 usage: 81.03 reweight: 1.00000 osd.145 size: 1.6 usage: 81.19 reweight: 1.00000 osd.137 size: 1.6 usage: 81.20 reweight: 0.89999 osd.136 size: 1.6 usage: 81.21 reweight: 0.89999 osd.54 size: 1.6 usage: 82.88 reweight: 1.00000 osd.252 size: 1.6 usage: 83.04 reweight: 0.89999 Question: Why is the data distribution on the 1.6TB disks unequal? How can I correct this? THX

4 10

rgw recovering shards
by Frank R 25 Oct '19

25 Oct '19

Hi all, After an RGW upgrade from 12.2.7 to 12.2.12 for RGW multisite a few days ago the "sync status" has constantly shown a few "recovering shards", ie: ----- # radosgw-admin sync status realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod) zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod) zone ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shards behind shards: [48] oldest incremental change not applied: 2019-10-21 22:34:11.0.293798s 5 shards are recovering recovering shards: [11,37,48,110,117] ----- This is the secondary zone. I am worried about the "oldest incremental change not applied" being from the 21st. Is there a way to have RGW just stop trying to recover these shards and just sync them from this point in time? thx Frank

2 1

Change device class in EC profile
by Frank Schilder 24 Oct '19

24 Oct '19

I recently moved an EC pool from HDD to SSD by changing the device class in the crush rule. I would like to complete this operation by cleaning up a dirty trail. The EC profile attached to this pool is called sr-ec-6-2-hdd and it is easy enough to rename that to sr-ec-6-2-ssd. However, the profile itself contains the device class as well: crush-device-class=hdd [...] I can already see the confusion this will cause in the future. Is this situation one of the few instances where using the --force option is warranted to change the device class of a profile, as in: osd erasure-code-profile set sr-ec-6-2-ssd crush-device-class=ssd --force If not, how can I change the device class of the profile? Many thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

Re: Don't know how to use bucket notification
by Yuval Lifshitz 24 Oct '19

24 Oct '19

the endpoint is not the RGW endpoint, it is the server to which you want to send the bucket notifications to. E.g. if you have a rabbitmq server running at address: 1.2.3.4, you should use: push-endpoint=amqp://1.2.3.4 note that in such a case the: amqp-exchange parameter must be set as well. assuming you have an http server, on the same address, listening on port 8080, and you want your notifications to get to it, you should use: push-endpoint=http://1.2.3.4:8080 Yuval > Subject: [ceph-users] Don't know how to use bucket notification > Date: Thu, 24 Oct 2019 16:26:54 +0800 > From: 柯名澤 <mingze.ke(a)gmail.com> > To: ceph-users(a)ceph.io > > > > Hi, all. > Does anyone know where the endpoint of CREATE TOPIC is? (for bucket > notification) > https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic > Is that the same with the normal S3 API? I tried but failed. > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

[ceph-user] Ceph mimic support FIPS
by Amit Ghadge 24 Oct '19

24 Oct '19

Hi all, We have FIPS enable cluster where it is running on ceph-12.2.12, after upgrading to mimic 13.2.6 can't serve any requests. and not able to get/put objects, buckets. Is there Mimic support FIPS? Thanks, Amit G

1 0

Don't know how to use bucket notification
by 柯名澤 24 Oct '19

24 Oct '19

Hi, all. Does anyone know where the endpoint of CREATE TOPIC is? (for bucket notification) https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic Is that the same with the normal S3 API? I tried but failed. Thanks.

1 0

PG badly corrupted after merging PGs on mixed FileStore/BlueStore setup
by Paul Emmerich 23 Oct '19

23 Oct '19

Hi, I'm working on a curious case that looks like a bug in PG merging maybe related to FileStore. Setup is 14.2.1 that is half BlueStore half FileStore (being migrated), and the number of PGs on an RGW index pool were reduced, now one of the PGs (3 FileStore OSDs) seems to be corrupted. There are some (29) objects that are affected (~20% of the PG), the issue looks like this for one of the affected objects which I'll call dir.A here # object seems to exist according to rados rados -p default.rgw.buckets.index ls | grep .dir.A .dir.A # or doesn't it? rados -p default.rgw.buckets.index get .dir.A - error getting default.rgw.buckets.index/.dir.A: (2) No such file or directory Running deep-scrub reports that everything is okay with the affected PG This is what the OSD logs when trying to access it, nothing really relevant with debug 20: 10 osd.57 pg_epoch: 1149030 pg[18.2( v 1148996'1422066 (1144429'1418988,1148996'1422066] local-lis/les=1149021/1149022 n=135 ec=49611/596 lis/c 1149021/1149021 les/c/f 1149022/1149022/0 1149015/1149021/1149021) [57,0,31] r=0 lpr=1149021 crt=1148996'1422066 lcod 1148996'1422065 mlcod 0'0 active+clean] get_object_context: no obc for soid 18:764060e4:::.dir.A:head and !can_create So going one level deeper with ceph-objectstore-tool: # --op list (29 messages like this) error getting default.rgw.buckets.index/.dir.A: (2) No such file or directory followed by a complete autoput of the json for the objects including the broken ones # .dir.A dump dump Error stat on : 18.2_head,#18:73996afb:::.dir.A:head#, (2) No such file or directory Error getting snapset on : 18.2_head,#18:73996afb:::.dir.A:head#, (2) No such file or directory { "id": { "oid": ".dir.A", "key": "", "snapid": -2, "hash": 3746994638, "max": 0, "pool": 18, "namespace": "", "max": 0 } } # --op export stops after encountering a bad object with 'export_files error -2' This is the same for all 3 OSDs in that PG. Has anyone encountered something similar? I'll probably just nuke the affected bucket indices tomorrow and re-create them. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

2 2