All;
We're setting up our second cluster, using version 14.2.4, and we've run into a weird issue: all of our OSDs are created with a size of 0 B. Weights are appropriate for the size of the underlying drives, but ceph -s shows this:
cluster:
id: <id>
health: HEALTH_WARN
Reduced data availability: 256 pgs inactive
too few PGs per OSD (28 < min 30)
services:
mon: 3 daemons, quorum s700041,s700042,s700043 (age 4d)
mgr: s700041(active, since 3d), standbys: s700042, s700043
osd: 9 osds: 9 up (since 21m), 9 in (since 44m)
data:
pools: 1 pools, 256 pgs
objects: 0 objects, 0 B
-->usage: 0 B used, 0 B / 0 B avail<-- (emphasis added)
pgs: 100.000% pgs unknown
256 unknown
Thoughts?
I have ceph-volumne.log, and the log from one of the OSD daemons, though it looks like the auth keys get printed to the ceph-volume.log.
Thank you,
Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
DHilsbos(a)PerformAir.com
300 S. Hamilton Pl.
Gilbert, AZ 85233
Phone: (480) 610-3500
Fax: (480) 610-3501
www.PerformAir.com
All;
We're setting up our second cluster, using version 14.2.4, and we've run into a weird issue: all of our OSDs are created with a size of 0 B. Weights are appropriate for the size of the underlying drives, but ceph -s shows this:
cluster:
id: <id>
health: HEALTH_WARN
Reduced data availability: 256 pgs inactive
too few PGs per OSD (28 < min 30)
services:
mon: 3 daemons, quorum s700041,s700042,s700043 (age 4d)
mgr: s700041(active, since 3d), standbys: s700042, s700043
osd: 9 osds: 9 up (since 21m), 9 in (since 44m)
data:
pools: 1 pools, 256 pgs
objects: 0 objects, 0 B
-->usage: 0 B used, 0 B / 0 B avail<-- (emphasis added)
pgs: 100.000% pgs unknown
256 unknown
Thoughts?
I have ceph-volumne.log, and the log from one of the OSD daemons, though it looks like the auth keys get printed to the ceph-volume.log.
Thank you,
Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
Hi all,
I have 40 nvme drives with about 20G free space each.
Would creating a 10GB partition/lvm on each of the nvmes for an rgw index
pool be a bad idea?
RGW has about about 5 million objects
I don't think space will be an issue but I am worried about the 10G size,
is it just too small for a bluestore OSD?
thx
Frank
++ceph-users
On Fri, Sep 20, 2019 at 4:48 PM Biswajeet Patra <
biswajeet.patra(a)flipkart.com> wrote:
> Hi,
> We recently faced an issue with radosgw authentication of presigned urls.
> The presigned url generated by client to download object fails at radosgw
> with http error 403 i.e SignatureDoesNotMatch.
>
> The radosgw computes the signature (*v2 signature in this case*) using
> the s3 specification
> https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html.
> In this process, a string_to_sign is created by concatenating selected
> elements from requests which is then authenticated using hmac to create the
> final signature. If this signature matches with the client signature, the
> authentication is successful or else it fails with SignatureDoesNotMatch
> error. As part of the StringToSign, the CanonicalizedAmzHeaders should only
> include headers that start with "x-amz-" and ignore other headers. But in
> the radosgw code, there are other meta prefixes that are checked against
> the http request headers and if matched are included in the
> CanonicalizedAmzHeaders to compute the final signature. For e.g, if a
> request header contains "HTTP_X_ACCOUNT" its selected by radosgw to include
> in amz_headers but the same will be ignored by AWS SDK as it does not start
> with "x-amz-". This will result in different signature computed by client
> and radosgw.
>
> Code Snippet: rgw_common.cc
> struct str_len meta_prefixes[] = { STR_LEN_ENTRY("HTTP_X_AMZ"),
> STR_LEN_ENTRY("HTTP_X_GOOG"),
> STR_LEN_ENTRY("HTTP_X_DHO"),
> STR_LEN_ENTRY("HTTP_X_RGW"),
> STR_LEN_ENTRY("HTTP_X_OBJECT"),
> STR_LEN_ENTRY("HTTP_X_CONTAINER"),
> STR_LEN_ENTRY("HTTP_X_ACCOUNT"),
> {NULL, 0} };
>
> The method init_meta_info() which matches the above prefixes is called
> from RGWREST::preprocess() which is invoked for all s3 requests. It will be
> helpful to know as to why these prefixes that are not specified by AWS S3
> comes in the path of s3 authentication. Was it added for swift use-cases
> only? If so, then why its included in rgw_common.cc?
> As a proposed fix, we can remove the highlighted meta prefixes that are
> not specified by aws from the s3 authentication path signature calculation.
> Let me know if you have any queries or solutions.
>
> Regards,
> Biswajeet
>
--
*-----------------------------------------------------------------------------------------*
*This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error, please notify the
system manager. This message contains confidential information and is
intended only for the individual named. If you are not the named addressee,
you should not disseminate, distribute or copy this email. Please notify
the sender immediately by email if you have received this email by mistake
and delete this email from your system. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this information is
strictly prohibited.*****
****
*Any views or opinions presented in this
email are solely those of the author and do not necessarily represent those
of the organization. Any information on shares, debentures or similar
instruments, recommended product pricing, valuations and the like are for
information purposes only. It is not meant to be an instruction or
recommendation, as the case may be, to buy or to sell securities, products,
services nor an offer to buy or sell securities, products or services
unless specifically stated to be so on behalf of the Flipkart group.
Employees of the Flipkart group of companies are expressly required not to
make defamatory statements and not to infringe or authorise any
infringement of copyright or any other legal right by email communications.
Any such communication is contrary to organizational policy and outside the
scope of the employment of the individual concerned. The organization will
not accept any liability in respect of such communication, and the employee
responsible will be personally liable for any damages or other liability
arising.*****
****
*Our organization accepts no liability for the
content of this email, or for the consequences of any actions taken on the
basis of the information *provided,* unless that information is
subsequently confirmed in writing. If you are not the intended recipient,
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited.*
_-----------------------------------------------------------------------------------------_
Hi all,
After an RGW upgrade from 12.2.7 to 12.2.12 for RGW multisite a few days
ago the "sync status" has constantly shown a few "recovering shards", ie:
-----
# radosgw-admin sync status
realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod)
zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod)
zone ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 1 shards
behind shards: [48]
oldest incremental change not applied: 2019-10-21
22:34:11.0.293798s
5 shards are recovering
recovering shards: [11,37,48,110,117]
-----
This is the secondary zone. I am worried about the "oldest incremental
change not applied" being from the 21st. Is there a way to have RGW just
stop trying to recover these shards and just sync them from this point in
time?
thx
Frank
I recently moved an EC pool from HDD to SSD by changing the device class in the crush rule. I would like to complete this operation by cleaning up a dirty trail. The EC profile attached to this pool is called sr-ec-6-2-hdd and it is easy enough to rename that to sr-ec-6-2-ssd. However, the profile itself contains the device class as well:
crush-device-class=hdd
[...]
I can already see the confusion this will cause in the future. Is this situation one of the few instances where using the --force option is warranted to change the device class of a profile, as in:
osd erasure-code-profile set sr-ec-6-2-ssd crush-device-class=ssd --force
If not, how can I change the device class of the profile?
Many thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
the endpoint is not the RGW endpoint, it is the server to which you
want to send the bucket notifications to.
E.g. if you have a rabbitmq server running at address: 1.2.3.4, you should use:
push-endpoint=amqp://1.2.3.4
note that in such a case the: amqp-exchange parameter must be set as well.
assuming you have an http server, on the same address, listening on
port 8080, and you want your notifications to get to it, you should
use:
push-endpoint=http://1.2.3.4:8080
Yuval
> Subject: [ceph-users] Don't know how to use bucket notification
> Date: Thu, 24 Oct 2019 16:26:54 +0800
> From: 柯名澤 <mingze.ke(a)gmail.com>
> To: ceph-users(a)ceph.io
>
>
>
> Hi, all.
> Does anyone know where the endpoint of CREATE TOPIC is? (for bucket
> notification)
> https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic
> Is that the same with the normal S3 API? I tried but failed.
> Thanks.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi all,
We have FIPS enable cluster where it is running on ceph-12.2.12, after
upgrading to mimic 13.2.6 can't serve any requests. and not able to get/put
objects, buckets.
Is there Mimic support FIPS?
Thanks,
Amit G
