- ceph-users - lists.ceph.io

Static website hosting with RGW
by Oliver Freyermuth 29 Oct '19

29 Oct '19

Dear Cephers, I have a question concerning static websites with RGW. To my understanding, it is best to run >=1 RGW client for "classic" S3 and in addition operate >=1 RGW client for website serving (potentially with HAProxy or its friends in front) to prevent messup of requests via the different protocols. I'd prefer to avoid "*.example.com" entries in DNS if possible. So my current setup has these settings for the "web" RGW client: rgw_enable_static_website = true rgw_enable_apis = s3website rgw_dns_s3website_name = some_value_unused_when_A_records_are_used_pointing_to_the_IP_but_it_needs_to_be_set and I create simple A records for each website pointing to the IP of this "web" RGW node. I can easily upload content for those websites to the other RGW instances which are serving S3, so S3 and s3website APIs are cleanly separated in separate instances. However, one issue remains: How do I run s3cmd ws-create on each website-bucket once? I can't do that against the "classic" S3-serving RGW nodes. This will give me a 405 (not allowed), since they do not have rgw_enable_static_website enabled. I also can not run it against the "web S3" nodes, since they do not have the S3 API enabled. Of course I could enable that, but then the RGW node can't cleanly disentangle S3 and website requests since I use A records. Does somebody have a good idea on how to solve this issue? Setting "rgw_enable_static_website = true" on the S3-serving RGW nodes would solve it, but does that have any bad side-effects on their S3 operation? Also, if there's an expert on this: Exposing a bucket under a tenant as static website is not possible since the colon (:) can't be encoded in DNS, right? In case somebody also wants to set something like this up, here are the best docs I could find: https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c and of course: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html-s… Cheers, Oliver

5 6

Help
by Sumit Gaur 28 Oct '19

28 Oct '19

Unsubscribe please from all email ids On Tue, 29 Oct 2019 at 7:12 am, <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Help (Sumit Gaur) > > > ---------------------------------------------------------------------- > > Date: Tue, 29 Oct 2019 07:06:17 +1100 > From: Sumit Gaur <sumitkgaur(a)gmail.com> > Subject: [ceph-users] Help > To: ceph-users(a)ceph.io > Message-ID: > <CAH_rbop2r7d4BVMi_cxYQgOaEcnpcdyjbsKR= > UdQwc9wBnAh_g(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000000dfea80595fe0971" > > --0000000000000dfea80595fe0971 > Content-Type: text/plain; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > On Tue, 29 Oct 2019 at 1:50 am, <ceph-users-request(a)ceph.io> wrote: > > > Send ceph-users mailing list submissions to > > ceph-users(a)ceph.io > > > > To subscribe or unsubscribe via email, send a message with subject or > > body 'help' to > > ceph-users-request(a)ceph.io > > > > You can reach the person managing the list at > > ceph-users-owner(a)ceph.io > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of ceph-users digest..." > > > > Today's Topics: > > > > 1. Re: subtrees have overcommitted (target_size_bytes / > > target_size_ratio) > > (Lars T=C3=A4uber) > > 2. After delete 8.5M Objects in a bucket still 500K left > > (EDH - Manuel Rios Fernandez) > > 3. Re: Static website hosting with RGW (Casey Bodley) > > > > > > ---------------------------------------------------------------------- > > > > Date: Mon, 28 Oct 2019 11:24:54 +0100 > > From: Lars T=C3=A4uber <taeuber(a)bbaw.de> > > Subject: [ceph-users] Re: subtrees have overcommitted > > (target_size_bytes / target_size_ratio) > > To: ceph-users <ceph-users(a)ceph.io> > > Message-ID: <20191028112454.0362fe66(a)bbaw.de> > > Content-Type: text/plain; charset=3DUTF-8 > > > > Is there a way to get rid of this warnings with activated autoscaler > > besides adding new osds? > > > > Yet I couldn't get a satisfactory answer to the question why this all > > happens. > > > > ceph osd pool autoscale-status : > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET > > RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > cephfs_data 122.2T 1.5 165.4T 1.1085 > > 0.8500 1.0 1024 on > > > > versus > > > > ceph df : > > RAW STORAGE: > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95 > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED > > MAX AVAIL > > cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 > > 12 TiB > > > > > > It seems that the overcommitment is wrongly calculated. Isn't the RATE > > already used to calculate the SIZE? > > > > It seems USED(df) =3D SIZE(autoscale-status) > > Isn't the RATE already taken into account here? > > > > Could someone please explain the numbers to me? > > > > > > Thanks! > > Lars > > > > Fri, 25 Oct 2019 07:42:58 +0200 > > Lars T=C3=A4uber <taeuber(a)bbaw.de> =3D=3D> Nathan Fish > <lordcirth(a)gmail.c= > om> : > > > Hi Nathan, > > > > > > Thu, 24 Oct 2019 10:59:55 -0400 > > > Nathan Fish <lordcirth(a)gmail.com> =3D=3D> Lars T=C3=A4uber > <taeuber@bba= > w.de> : > > > > Ah, I see! The BIAS reflects the number of placement groups it should > > > > create. Since cephfs metadata pools are usually very small, but have > > > > many objects and high IO, the autoscaler gives them 4x the number of > > > > placement groups that it would normally give for that amount of data. > > > > > > > ah ok, I understand. > > > > > > > So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to > > > > 0.3? Are the two pools using entirely different device classes, so > > > > they are not sharing space? > > > > > > Yes, the metadata is on SSDs and the data on HDDs. > > > > > > > Anyway, I see that your overcommit is only "1.031x". So if you set > > > > cephfs_data to 0.85, it should go away. > > > > > > This is not the case. I set the target_ratio to 0.7 and get this: > > > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO > > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > cephfs_metadata 15736M 3.0 2454G 0.0188 > > 0.3000 4.0 256 on > > > cephfs_data 122.2T 1.5 165.4T 1.1085 > > 0.7000 1.0 1024 on > > > > > > The ratio seems to have nothing to do with the target_ratio but the > SIZ= > E > > and the RAW_CAPACITY. > > > Because the pool is still getting more data the SIZE increases and > > therefore the RATIO increases. > > > The RATIO seems to be calculated by this formula > > > RATIO =3D SIZE * RATE / RAW_CAPACITY. > > > > > > This is what I don't understand. The data in the cephfs_data pool seems > > to need more space than the raw capacity of the cluster provides. Hence > t= > he > > situation is called "overcommitment". > > > > > > But why is this only the case when the autoscaler is active? > > > > > > Thanks > > > Lars > > > > > > > > > > > On Thu, Oct 24, 2019 at 10:09 AM Lars T=C3=A4uber <taeuber(a)bbaw.de> > > wrote: > > > > > > > > > > Thanks Nathan for your answer, > > > > > > > > > > but I set the the Target Ratio to 0.9. It is the cephfs_data pool > > that makes the troubles. > > > > > > > > > > The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is > > not explained on the page linked below. So I don't know its meaning. > > > > > > > > > > How can be a pool overcommited when it is the only pool on a set of > > OSDs? > > > > > > > > > > Best regards, > > > > > Lars > > > > > > > > > > Thu, 24 Oct 2019 09:39:51 -0400 > > > > > Nathan Fish <lordcirth(a)gmail.com> =3D=3D> Lars T=C3=A4uber > <taeuber= > @bbaw.de> > > : > > > > > > The formatting is mangled on my phone, but if I am reading it > > correctly, > > > > > > you have set Target Ratio to 4.0. This means you have told the > > balancer > > > > > > that this pool will occupy 4x the space of your whole cluster, > an= > d > > to > > > > > > optimize accordingly. This is naturally a problem. Setting it to > = > 0 > > will > > > > > > clear the setting and allow the autobalancer to work. > > > > > > > > > > > > On Thu., Oct. 24, 2019, 5:18 a.m. Lars T=C3=A4uber, <taeuber@bbaw > = > .de> > > wrote: > > > > > > > > > > > > > This question is answered here: > > > > > > > > https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning= > / > > > > > > > > > > > > > > But it tells me that there is more data stored in the pool than > > the raw > > > > > > > capacity provides (taking the replication factor RATE into > > account) hence > > > > > > > the RATIO being above 1.0 . > > > > > > > > > > > > > > How comes this is the case? - Data is stored outside of the > poo= > l? > > > > > > > How comes this is only the case when the autoscaler is active? > > > > > > > > > > > > > > Thanks > > > > > > > Lars > > > > > > > > > > > > > > > > > > > > > Thu, 24 Oct 2019 10:36:52 +0200 > > > > > > > Lars T=C3=A4uber <taeuber(a)bbaw.de> =3D=3D> ceph-users(a)ceph.io > : > > > > > > > > My question requires too complex an answer. > > > > > > > > So let me ask a simple question: > > > > > > > > > > > > > > > > What does the SIZE of "osd pool autoscale-status" > > tell/mean/comes from? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Lars > > > > > > > > > > > > > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > > > > > > > Lars T=C3=A4uber <taeuber(a)bbaw.de> =3D=3D> > ceph-users(a)ceph.io= > : > > > > > > > > > Hello everybody! > > > > > > > > > > > > > > > > > > What does this mean? > > > > > > > > > > > > > > > > > > health: HEALTH_WARN > > > > > > > > > 1 subtrees have overcommitted pool > > target_size_bytes > > > > > > > > > 1 subtrees have overcommitted pool > > target_size_ratio > > > > > > > > > > > > > > > > > > and what does it have to do with the autoscaler? > > > > > > > > > When I deactivate the autoscaler the warning goes away. > > > > > > > > > > > > > > > > > > > > > > > > > > > $ ceph osd pool autoscale-status > > > > > > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY > > RATIO > > > > > > > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > > > > > > > cephfs_metadata 15106M 3.0 2454G > > 0.0180 > > > > > > > 0.3000 4.0 256 on > > > > > > > > > cephfs_data 113.6T 1.5 165.4T > > 1.0306 > > > > > > > 0.9000 1.0 512 on > > > > > > > > > > > > > > > > > > > > > > > > > > > $ ceph health detail > > > > > > > > > HEALTH_WARN 1 subtrees have overcommitted pool > > target_size_bytes; 1 > > > > > > > subtrees have overcommitted pool target_size_ratio > > > > > > > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have > > overcommitted > > > > > > > pool target_size_bytes > > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > > 1.031x due > > > > > > > to target_size_bytes 0 on pools [] > > > > > > > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have > > overcommitted > > > > > > > pool target_size_ratio > > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > > 1.031x due > > > > > > > to target_size_ratio 0.900 on pools ['cephfs_data'] > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Lars > > > > > > > > > _______________________________________________ > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > ------------------------------ > > > > Date: Mon, 28 Oct 2019 14:18:01 +0100 > > From: "EDH - Manuel Rios Fernandez" <mriosfer(a)easydatahost.com> > > Subject: [ceph-users] After delete 8.5M Objects in a bucket still 500K > > left > > To: <ceph-users(a)ceph.io> > > Message-ID: <02a201d58d92$1fe85880$5fb90980$(a)easydatahost.com> > > Content-Type: multipart/alternative; > > boundary=3D"----=3D_NextPart_000_02A3_01D58D9A.81B17B70" > > > > This is a multipart message in MIME format. > > > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > > Content-Type: text/plain; > > charset=3D"us-ascii" > > Content-Transfer-Encoding: 7bit > > > > Hi Ceph's! > > > > > > > > We started deteling a bucket several days ago. Total size 47TB / 8.5M > > objects. > > > > > > > > Now we see the cli bucket rm stucked and by console drop this messages. > > > > > > > > [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700 0 > > abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart > uploa= > ds > > > > 2019-10-28 13:56:24.021 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 2000 incomplete multipart uploads > > > > 2019-10-28 13:57:04.726 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 3000 incomplete multipart uploads > > > > 2019-10-28 13:57:45.424 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 4000 incomplete multipart uploads > > > > 2019-10-28 13:58:25.905 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 5000 incomplete multipart uploads > > > > 2019-10-28 13:59:06.898 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 6000 incomplete multipart uploads > > > > 2019-10-28 13:59:47.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 7000 incomplete multipart uploads > > > > 2019-10-28 14:00:42.102 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 8000 incomplete multipart uploads > > > > 2019-10-28 14:01:23.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 9000 incomplete multipart uploads > > > > 2019-10-28 14:02:06.028 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 10000 incomplete multipart uploads > > > > 2019-10-28 14:02:48.648 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 11000 incomplete multipart uploads > > > > 2019-10-28 14:03:29.807 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 12000 incomplete multipart uploads > > > > 2019-10-28 14:04:11.180 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 13000 incomplete multipart uploads > > > > 2019-10-28 14:04:52.396 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 14000 incomplete multipart uploads > > > > 2019-10-28 14:05:33.050 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 15000 incomplete multipart uploads > > > > 2019-10-28 14:06:13.652 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 16000 incomplete multipart uploads > > > > 2019-10-28 14:06:54.806 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 17000 incomplete multipart uploads > > > > 2019-10-28 14:07:35.867 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 18000 incomplete multipart uploads > > > > 2019-10-28 14:08:16.886 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 19000 incomplete multipart uploads > > > > 2019-10-28 14:08:57.711 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 20000 incomplete multipart uploads > > > > 2019-10-28 14:09:38.032 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 21000 incomplete multipart uploads > > > > 2019-10-28 14:10:18.377 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 22000 incomplete multipart uploads > > > > 2019-10-28 14:10:58.833 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 23000 incomplete multipart uploads > > > > 2019-10-28 14:11:39.078 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 24000 incomplete multipart uploads > > > > 2019-10-28 14:12:24.731 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 25000 incomplete multipart uploads > > > > 2019-10-28 14:13:12.176 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 26000 incomplete multipart uploads > > > > > > > > > > > > Bucket stats show 500K objects left. Looks like bucket rm is trying to > > abort > > all incompleted mutipart. But in bucket stats this operation is not > > reflected removing objects from stats. > > > > > > > > May be wait to get up 500K or it's a bug? > > > > > > > > Regards > > > > Manuel > > > > > > > > > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > > Content-Type: text/html; > > charset=3D"us-ascii" > > Content-Transfer-Encoding: quoted-printable > > > > <html xmlns:v=3D3D"urn:schemas-microsoft-com:vml" =3D > > xmlns:o=3D3D"urn:schemas-microsoft-com:office:office" =3D > > xmlns:w=3D3D"urn:schemas-microsoft-com:office:word" =3D > > xmlns:m=3D3D"http://schemas.microsoft.com/office/2004/12/omml" =3D > > xmlns=3D3D"http://www.w3.org/TR/REC-html40"><head><meta =3D > > http-equiv=3D3DContent-Type content=3D3D"text/html; =3D > > charset=3D3Dus-ascii"><meta name=3D3DGenerator content=3D3D"Microsoft > Wor= > d 15 =3D > > (filtered medium)"><style></style></head><body lang=3D3DES =3D > > link=3D3D"#0563C1" vlink=3D3D"#954F72"><div class=3D3DWordSection1> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Hi =3D > > Ceph’s!<o:p></o:p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>We started deteling a = > =3D > > bucket several days ago. Total size 47TB / 8.5M =3D > > objects.<o:p></o:p> lang=3D3DEN-US= > =3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Now we see the cli =3D > > bucket rm stucked and by console drop this =3D > > messages.<o:p></o:p> lang=3D3DEN-U= > S =3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>[root@ceph-rgw03 ~]# = > =3D > > 2019-10-28 13:55:43.880 7f0dd92c9700  0 abort_bucket_multiparts =3D > > WARNING : aborted 1000 incomplete multipart =3D > > uploads<o:p></o:p> = > =3D > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:56:24.021= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 2000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:57:04.726= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 3000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:57:45.424= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 4000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:58:25.905= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 5000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:59:06.898= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 6000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:59:47.829= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 7000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:00:42.102= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 8000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:01:23.829= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 9000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:02:06.028= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 10000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:02:48.648= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 11000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:03:29.807= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 12000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:04:11.180= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 13000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:04:52.396= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 14000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:05:33.050= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 15000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:06:13.652= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 16000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:06:54.806= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 17000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:07:35.867= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 18000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:08:16.886= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 19000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:08:57.711= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 20000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:09:38.032= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 21000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:10:18.377= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 22000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:10:58.833= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 23000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:11:39.078= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 24000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:12:24.731= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 25000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:13:12.176= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 26000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Bucket stats show 500K > = > =3D > > objects left. Looks like bucket rm is trying to abort all incompleted =3D > > mutipart. But in bucket stats this operation is not reflected removing = > =3D > > objects from stats.<o:p></o:p> > lang=3D3DEN-US =3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>May be wait to get up = > =3D > > 500K or it’s a bug?<o:p></o:p> > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Regards<o:p></o:p></spa= > n>=3D > > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Manuel<o:p></o:p></span= > ><=3D > > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p></div></body></html> > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70-- > > > > ------------------------------ > > > > Date: Mon, 28 Oct 2019 10:48:44 -0400 > > From: Casey Bodley <cbodley(a)redhat.com> > > Subject: [ceph-users] Re: Static website hosting with RGW > > To: ceph-users(a)ceph.io > > Message-ID: <20834361-445e-1ee5-433b-dd4792f90608(a)redhat.com> > > Content-Type: text/plain; charset=3DUTF-8; format=3Dflowed > > > > > > On 10/24/19 8:38 PM, Oliver Freyermuth wrote: > > > Dear Cephers, > > > > > > I have a question concerning static websites with RGW. > > > To my understanding, it is best to run >=3D1 RGW client for "classic" > S= > 3 > > and in addition operate >=3D1 RGW client for website serving > > > (potentially with HAProxy or its friends in front) to prevent messup of > > requests via the different protocols. > > > > > > I'd prefer to avoid "*.example.com" entries in DNS if possible. > > > So my current setup has these settings for the "web" RGW client: > > > rgw_enable_static_website =3D true > > > rgw_enable_apis =3D s3website > > > rgw_dns_s3website_name =3D > > > some_value_unused_when_A_records_are_used_pointing_to_the_IP_but_it_needs= > _to_be_set > > > and I create simple A records for each website pointing to the IP of > > this "web" RGW node. > > > > > > I can easily upload content for those websites to the other RGW > > instances which are serving S3, > > > so S3 and s3website APIs are cleanly separated in separate instances. > > > > > > However, one issue remains: How do I run > > > s3cmd ws-create > > > on each website-bucket once? > > > I can't do that against the "classic" S3-serving RGW nodes. This will > > give me a 405 (not allowed), > > > since they do not have rgw_enable_static_website enabled. > > > I also can not run it against the "web S3" nodes, since they do not > hav= > e > > the S3 API enabled. > > > Of course I could enable that, but then the RGW node can't cleanly > > disentangle S3 and website requests since I use A records. > > > > > > Does somebody have a good idea on how to solve this issue? > > > Setting "rgw_enable_static_website =3D true" on the S3-serving RGW > node= > s > > would solve it, but does that have any bad side-effects on their S3 > > operation? > > > > Enabling static website on the gateway serving the S3 api does look like > > the right solution. As far as I can tell, it's only used to control > > whether the S3 ops for PutBucketWebsite, GetBucketWebsite, and > > DeleteBucketWebsite are exposed. > > > > > > > > Also, if there's an expert on this: Exposing a bucket under a tenant as > > static website is not possible since the colon (:) can't be encoded in > DN= > S, > > right? > > > > > > > > > In case somebody also wants to set something like this up, here are the > > best docs I could find: > > > https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c > > > and of course: > > > > > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html= > > -single/object_gateway_guide_for_red_hat_enterprise_linux/index#configuring= > <https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html=-…> > _gateways_for_static_web_hosting > > > > > > > > > Cheers, > > > Oliver > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > > > > ------------------------------ > > > > End of ceph-users Digest, Vol 81, Issue 79 > > ****************************************** > > > > --0000000000000dfea80595fe0971 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div> </div><div> <div class=3D"gmail_quote"><div dir=3D"ltr" class= > =3D"gmail_attr">On Tue, 29 Oct 2019 at 1:50 am, <<a href=3D"mailto: > ceph-= > users-request(a)ceph.io">ceph-users-request(a)ceph.io</a>> > wrote: </div><= > blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 > .8ex;border-left:1px= > #ccc solid;padding-left:1ex">Send ceph-users mailing list submissions > to<b= > r> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:ceph-users@ceph.io" > target=3D= > "_blank">ceph-users(a)ceph.io</a> > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:ceph-users-request@ceph.io" > t= > arget=3D"_blank">ceph-users-request(a)ceph.io</a> > > You can reach the person managing the list at > =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:ceph-users-owner@ceph.io" > tar= > get=3D"_blank">ceph-users-owner(a)ceph.io</a> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > =C2=A0 =C2=A01. Re: subtrees have overcommitted (target_size_bytes / > target= > _size_ratio) > =C2=A0 =C2=A0 =C2=A0 (Lars T=C3=A4uber) > =C2=A0 =C2=A02. After delete 8.5M Objects in a bucket still 500K left > =C2=A0 =C2=A0 =C2=A0 (EDH - Manuel Rios Fernandez) > =C2=A0 =C2=A03. Re: Static website hosting with RGW (Casey Bodley) > > > ---------------------------------------------------------------------- > > Date: Mon, 28 Oct 2019 11:24:54 +0100 > From: Lars T=C3=A4uber <<a href=3D"mailto:taeuber@bbaw.de" > target=3D"_bl= > ank">taeuber(a)bbaw.de</a>> > Subject: [ceph-users] Re: subtrees have overcommitted > =C2=A0 =C2=A0 =C2=A0 =C2=A0 (target_size_bytes / target_size_ratio) > To: ceph-users <<a href=3D"mailto:ceph-users@ceph.io" > target=3D"_blank">= > ceph-users(a)ceph.io</a>> > Message-ID: <<a href=3D"mailto:20191028112454.0362fe66@bbaw.de" target= > =3D"_blank">20191028112454.0362fe66(a)bbaw.de</a>> > Content-Type: text/plain; charset=3DUTF-8 > > Is there a way to get rid of this warnings with activated autoscaler > beside= > s adding new osds? > > Yet I couldn't get a satisfactory answer to the question why this all > h= > appens. > > ceph osd pool autoscale-status : > =C2=A0POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2=A0SIZE=C2=A0= > TARGET SIZE=C2=A0 RATE=C2=A0 RAW CAPACITY=C2=A0 =C2=A0RATIO=C2=A0 TARGET > R= > ATIO=C2=A0 BIAS=C2=A0 PG_NUM=C2=A0 NEW PG_NUM=C2=A0 AUTOSCALE > =C2=A0cephfs_data=C2=A0 =C2=A0 =C2=A0 122.2T=C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 1.5=C2=A0 =C2=A0 =C2=A0 =C2=A0 165.4T=C2=A0 > 1.1085= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 0.8500=C2=A0 =C2=A01.0=C2=A0 =C2=A0 1024=C2=A0 > = > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on > > versus > > =C2=A0ceph df=C2=A0 : > RAW STORAGE: > =C2=A0 =C2=A0 CLASS=C2=A0 =C2=A0 =C2=A0SIZE=C2=A0 =C2=A0 =C2=A0 =C2=A0 > AVAI= > L=C2=A0 =C2=A0 =C2=A0 =C2=A0USED=C2=A0 =C2=A0 =C2=A0 =C2=A0 RAW USED=C2=A0 > = > =C2=A0 =C2=A0%RAW USED > =C2=A0 =C2=A0 hdd=C2=A0 =C2=A0 =C2=A0 =C2=A0165 TiB=C2=A0 =C2=A0 =C2=A0 41 > = > TiB=C2=A0 =C2=A0 =C2=A0124 TiB=C2=A0 =C2=A0 =C2=A0 124 TiB=C2=A0 =C2=A0 > =C2= > =A0 =C2=A0 =C2=A074.95 > > POOLS: > =C2=A0 =C2=A0 POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > I= > D=C2=A0 =C2=A0 =C2=A0STORED=C2=A0 =C2=A0 =C2=A0OBJECTS=C2=A0 =C2=A0 > =C2=A0U= > SED=C2=A0 =C2=A0 =C2=A0 =C2=A0 %USED=C2=A0 =C2=A0 =C2=A0MAX AVAIL > =C2=A0 =C2=A0 cephfs_data=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2=A0 > = > =C2=A075 TiB=C2=A0 =C2=A0 =C2=A0 49.31M=C2=A0 =C2=A0 =C2=A0122 TiB=C2=A0 = > =C2=A0 =C2=A087.16=C2=A0 =C2=A0 =C2=A0 =C2=A0 12 TiB > > > It seems that the overcommitment is wrongly calculated. Isn't the RATE > = > already used to calculate the SIZE? > > It seems USED(df) =3D SIZE(autoscale-status) > Isn't the RATE already taken into account here? > > Could someone please explain the numbers to me? > > > Thanks! > Lars > > Fri, 25 Oct 2019 07:42:58 +0200 > Lars T=C3=A4uber <<a href=3D"mailto:taeuber@bbaw.de" > target=3D"_blank">t= > aeuber(a)bbaw.de</a>> =3D=3D> Nathan Fish <<a href=3D"mailto: > lordcir= > th(a)gmail.com" target=3D"_blank">lordcirth(a)gmail.com</a>> : > > Hi Nathan, > > > > Thu, 24 Oct 2019 10:59:55 -0400 > > Nathan Fish <<a href=3D"mailto:lordcirth@gmail.com" > target=3D"_blan= > k">lordcirth(a)gmail.com</a>> =3D=3D> Lars T=C3=A4uber <<a > href=3D"m= > ailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> : > > > Ah, I see! The BIAS reflects the number of placement groups it > sh= > ould > > > create. Since cephfs metadata pools are usually very small, but > h= > ave > > > many objects and high IO, the autoscaler gives them 4x the > number= > of > > > placement groups that it would normally give for that amount of > d= > ata. > > >=C2=A0 =C2=A0 > > ah ok, I understand. > > > > > So, your cephfs_data is set to a ratio of 0.9, and > cephfs_metadat= > a to > > > 0.3? Are the two pools using entirely different device classes, > s= > o > > > they are not sharing space?=C2=A0 > > > > Yes, the metadata is on SSDs and the data on HDDs. > > > > > Anyway, I see that your overcommit is only "1.031x". > So= > if you set > > > cephfs_data to 0.85, it should go away.=C2=A0 > > > > This is not the case. I set the target_ratio to 0.7 and get this: > > > >=C2=A0 POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE= > =C2=A0 TARGET SIZE=C2=A0 RATE=C2=A0 RAW CAPACITY=C2=A0 =C2=A0RATIO=C2=A0 > TA= > RGET RATIO=C2=A0 BIAS=C2=A0 PG_NUM=C2=A0 NEW PG_NUM=C2=A0 AUTOSCALE > >=C2=A0 cephfs_metadata=C2=A0 15736M=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = > =C2=A0 =C2=A0 =C2=A0 3.0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02454G=C2=A0 > 0.018= > 8=C2=A0 =C2=A0 =C2=A0 =C2=A0 0.3000=C2=A0 =C2=A04.0=C2=A0 =C2=A0 =C2=A0256= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 > =C2= > =A0 > >=C2=A0 cephfs_data=C2=A0 =C2=A0 =C2=A0 122.2T=C2=A0 =C2=A0 =C2=A0 =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.5=C2=A0 =C2=A0 =C2=A0 =C2=A0 > 165.4T=C2=A0= > 1.1085=C2=A0 =C2=A0 =C2=A0 =C2=A0 0.7000=C2=A0 =C2=A01.0=C2=A0 =C2=A0 > 1024= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 > =C2= > =A0 > > > > The ratio seems to have nothing to do with the target_ratio but the > SI= > ZE and the RAW_CAPACITY. > > Because the pool is still getting more data the SIZE increases and > the= > refore the RATIO increases. > > The RATIO seems to be calculated by this formula > > RATIO =3D SIZE * RATE / RAW_CAPACITY. > > > > This is what I don't understand. The data in the cephfs_data pool > = > seems to need more space than the raw capacity of the cluster provides. > Hen= > ce the situation is called "overcommitment". > > > > But why is this only the case when the autoscaler is active? > > > > Thanks > > Lars > > > > > > > > On Thu, Oct 24, 2019 at 10:09 AM Lars T=C3=A4uber <<a > href=3D"= > mailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> > wrote:=C2= > =A0 > > > > > > > > Thanks Nathan for your answer, > > > > > > > > but I set the the Target Ratio to 0.9. It is the > cephfs_data= > pool that makes the troubles. > > > > > > > > The 4.0 is the BIAS from the cephfs_metadata pool. This > &quo= > t;BIAS" is not explained on the page linked below. So I don't > know= > its meaning. > > > > > > > > How can be a pool overcommited when it is the only pool on > a= > set of OSDs? > > > > > > > > Best regards, > > > > Lars > > > > > > > > Thu, 24 Oct 2019 09:39:51 -0400 > > > > Nathan Fish <<a href=3D"mailto:lordcirth@gmail.com" > targe= > t=3D"_blank">lordcirth(a)gmail.com</a>> =3D=3D> Lars T=C3=A4uber > <<a= > href=3D"mailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> > = > :=C2=A0 =C2=A0 > > > > > The formatting is mangled on my phone, but if I am > read= > ing it correctly, > > > > > you have set Target Ratio to 4.0. This means you have > t= > old the balancer > > > > > that this pool will occupy 4x the space of your whole > c= > luster, and to > > > > > optimize accordingly. This is naturally a problem. > Sett= > ing it to 0 will > > > > > clear the setting and allow the autobalancer to > work.<b= > r> > > > > > > > > > > On Thu., Oct. 24, 2019, 5:18 a.m. Lars T=C3=A4uber, > &lt= > ;<a href=3D"mailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de > </a>&g= > t; wrote: > > > > >=C2=A0 =C2=A0 > > > > > > This question is answered here: > > > > > > <a href=3D" > https://ceph.io/rados/new-in-nautilus-p= > g-merging-and-autotuning/ > <https://ceph.io/rados/new-in-nautilus-p=g-merging-and-autotuning/>" > rel=3D"noreferrer" target=3D"_blank">https://cep= > h.io/rados/new-in-nautilus-pg-merging-and-autotuning/</a> > > > > > > > > > > > > But it tells me that there is more data stored in > = > the pool than the raw > > > > > > capacity provides (taking the replication factor > R= > ATE into account) hence > > > > > > the RATIO being above 1.0 . > > > > > > > > > > > > How comes this is the case? - Data is stored > outsi= > de of the pool? > > > > > > How comes this is only the case when the > autoscale= > r is active? > > > > > > > > > > > > Thanks > > > > > > Lars > > > > > > > > > > > > > > > > > > Thu, 24 Oct 2019 10:36:52 +0200 > > > > > > Lars T=C3=A4uber <<a href=3D"mailto: > taeuber@bba= > w.de" target=3D"_blank">taeuber(a)bbaw.de</a>> =3D=3D> <a > href=3D"mailt= > o:ceph-users@ceph.io" target=3D"_blank">ceph-users(a)ceph.io</a> :=C2=A0 > =C2= > =A0 > > > > > > > My question requires too complex an > answer.<b= > r> > > > > > > > So let me ask a simple question: > > > > > > > > > > > > > > What does the SIZE of "osd pool > autoscal= > e-status" tell/mean/comes from? > > > > > > > > > > > > > > Thanks > > > > > > > Lars > > > > > > > > > > > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > > > > > > Lars T=C3=A4uber <<a href=3D"mailto: > taeube= > r(a)bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> =3D=3D> <a > href=3D"= > mailto:ceph-users@ceph.io" target=3D"_blank">ceph-users(a)ceph.io</a> > :=C2=A0= > =C2=A0 > > > > > > > > Hello everybody! > > > > > > > > > > > > > > > > What does this mean? > > > > > > > > > > > > > > > >=C2=A0 =C2=A0 =C2=A0health: > HEALTH_WARN<b= > r> > > > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2=A0= > =C2=A01 subtrees have overcommitted pool target_size_bytes > > > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2=A0= > =C2=A01 subtrees have overcommitted pool target_size_ratio > > > > > > > > > > > > > > > > and what does it have to do with the > aut= > oscaler? > > > > > > > > When I deactivate the autoscaler the > war= > ning goes away. > > > > > > > > > > > > > > > > > > > > > > > > $ ceph osd pool autoscale-status > > > > > > > >=C2=A0 POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 = > =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE=C2=A0 TARGET SIZE=C2=A0 RATE=C2=A0 RAW > CAPA= > CITY=C2=A0 =C2=A0RATIO=C2=A0 =C2=A0 > > > > > > TARGET RATIO=C2=A0 BIAS=C2=A0 PG_NUM=C2=A0 NEW > PG_= > NUM=C2=A0 AUTOSCALE=C2=A0 =C2=A0 > > > > > > > >=C2=A0 cephfs_metadata=C2=A0 > 15106M=C2=A0= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3.0=C2=A0 =C2=A0 =C2=A0 = > =C2=A0 =C2=A02454G=C2=A0 0.0180=C2=A0 =C2=A0 > > > > > >=C2=A0 =C2=A00.3000=C2=A0 =C2=A04.0=C2=A0 =C2=A0 = > =C2=A0256=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 > <= > br> > > > > > > > >=C2=A0 cephfs_data=C2=A0 =C2=A0 =C2=A0 > 11= > 3.6T=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.5=C2=A0 =C2= > =A0 =C2=A0 =C2=A0 165.4T=C2=A0 1.0306=C2=A0 =C2=A0 > > > > > >=C2=A0 =C2=A00.9000=C2=A0 =C2=A01.0=C2=A0 =C2=A0 = > =C2=A0512=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 > <= > br> > > > > > > > > > > > > > > > > > > > > > > > > $ ceph health detail > > > > > > > > HEALTH_WARN 1 subtrees have > overcommitte= > d pool target_size_bytes; 1=C2=A0 =C2=A0 > > > > > > subtrees have overcommitted pool > target_size_ratio= > =C2=A0 =C2=A0 > > > > > > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 > s= > ubtrees have overcommitted=C2=A0 =C2=A0 > > > > > > pool target_size_bytes=C2=A0 =C2=A0 > > > > > > > >=C2=A0 =C2=A0 =C2=A0Pools > ['cephfs_da= > ta'] overcommit available storage by 1.031x due=C2=A0 =C2=A0 > > > > > > to target_size_bytes=C2=A0 =C2=A0 0=C2=A0 on > pools= > []=C2=A0 =C2=A0 > > > > > > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 > s= > ubtrees have overcommitted=C2=A0 =C2=A0 > > > > > > pool target_size_ratio=C2=A0 =C2=A0 > > > > > > > >=C2=A0 =C2=A0 =C2=A0Pools > ['cephfs_da= > ta'] overcommit available storage by 1.031x due=C2=A0 =C2=A0 > > > > > > to target_size_ratio 0.900 on pools > ['cephfs_d= > ata']=C2=A0 =C2=A0 > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > Lars > > > > > > > > > ________________________________________= > _______ > > > > > > > > ceph-users mailing list -- <a > href=3D"ma= > ilto:ceph-users@ceph.io" target=3D"_blank">ceph-users(a)ceph.io</a> > > > > > > > > To unsubscribe send an email to <a href= > =3D"mailto:ceph-users-leave@ceph.io" > target=3D"_blank">ceph-users-leave@cep= > h.io</a>=C2=A0 =C2=A0 > > ------------------------------ > > Date: Mon, 28 Oct 2019 14:18:01 +0100 > From: "EDH - Manuel Rios Fernandez" <<a href=3D"mailto: > mriosfe= > r(a)easydatahost.com" target=3D"_blank">mriosfer(a)easydatahost.com > </a>> > Subject: [ceph-users] After delete 8.5M Objects in a bucket still 500K > =C2=A0 =C2=A0 =C2=A0 =C2=A0 left > To: <<a href=3D"mailto:ceph-users@ceph.io" > target=3D"_blank">ceph-users@= > ceph.io</a>> > Message-ID: <02a201d58d92$1fe85880$5fb90980$@<a href=3D" > http://easydatah= > ost.com" rel=3D"noreferrer" target=3D"_blank">easydatahost.com</a>> > Content-Type: multipart/alternative; > =C2=A0 =C2=A0 =C2=A0 =C2=A0 > boundary=3D"----=3D_NextPart_000_02A3_01D5= > 8D9A.81B17B70" > > This is a multipart message in MIME format. > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/plain; > =C2=A0 =C2=A0 =C2=A0 =C2=A0 charset=3D"us-ascii" > Content-Transfer-Encoding: 7bit > > Hi Ceph's! > > > > We started deteling a bucket several days ago. Total size 47TB / 8.5M > objects. > > > > Now we see the cli bucket rm stucked and by console drop this messages. > > > > [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700=C2=A0 0 > abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart > uploads= > > > 2019-10-28 13:56:24.021 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 2000 incomplete multipart uploads > > 2019-10-28 13:57:04.726 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 3000 incomplete multipart uploads > > 2019-10-28 13:57:45.424 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 4000 incomplete multipart uploads > > 2019-10-28 13:58:25.905 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 5000 incomplete multipart uploads > > 2019-10-28 13:59:06.898 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 6000 incomplete multipart uploads > > 2019-10-28 13:59:47.829 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 7000 incomplete multipart uploads > > 2019-10-28 14:00:42.102 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 8000 incomplete multipart uploads > > 2019-10-28 14:01:23.829 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 9000 incomplete multipart uploads > > 2019-10-28 14:02:06.028 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 10000 incomplete multipart uploads > > 2019-10-28 14:02:48.648 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 11000 incomplete multipart uploads > > 2019-10-28 14:03:29.807 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 12000 incomplete multipart uploads > > 2019-10-28 14:04:11.180 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 13000 incomplete multipart uploads > > 2019-10-28 14:04:52.396 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 14000 incomplete multipart uploads > > 2019-10-28 14:05:33.050 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 15000 incomplete multipart uploads > > 2019-10-28 14:06:13.652 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 16000 incomplete multipart uploads > > 2019-10-28 14:06:54.806 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 17000 incomplete multipart uploads > > 2019-10-28 14:07:35.867 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 18000 incomplete multipart uploads > > 2019-10-28 14:08:16.886 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 19000 incomplete multipart uploads > > 2019-10-28 14:08:57.711 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 20000 incomplete multipart uploads > > 2019-10-28 14:09:38.032 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 21000 incomplete multipart uploads > > 2019-10-28 14:10:18.377 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 22000 incomplete multipart uploads > > 2019-10-28 14:10:58.833 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 23000 incomplete multipart uploads > > 2019-10-28 14:11:39.078 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 24000 incomplete multipart uploads > > 2019-10-28 14:12:24.731 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 25000 incomplete multipart uploads > > 2019-10-28 14:13:12.176 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 26000 incomplete multipart uploads > > > > > > Bucket stats show 500K objects left. Looks like bucket rm is trying to > abor= > t > all incompleted mutipart. But in bucket stats this operation is not > reflected removing objects from stats. > > > > May be wait to get up 500K or it's a bug? > > > > Regards > > Manuel > > > > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/html; > =C2=A0 =C2=A0 =C2=A0 =C2=A0 charset=3D"us-ascii" > Content-Transfer-Encoding: quoted-printable > > <html xmlns:v=3D3D"urn:schemas-microsoft-com:vml" =3D > xmlns:o=3D3D"urn:schemas-microsoft-com:office:office" =3D > xmlns:w=3D3D"urn:schemas-microsoft-com:office:word" =3D > xmlns:m=3D3D"<a href=3D" > http://schemas.microsoft.com/office/2004/12/om= > ml" rel=3D"noreferrer" target=3D"_blank"> > http://schemas.microsoft.com/offic= > e/2004/12/omml <http://schemas.microsoft.com/offic=e/2004/12/omml></a>" > =3D > xmlns=3D3D"<a href=3D"http://www.w3.org/TR/REC-html40" > rel=3D"noreferr= > er" target=3D"_blank">http://www.w3.org/TR/REC-html40 > </a>"><head= > ><meta =3D > http-equiv=3D3DContent-Type content=3D3D"text/html; =3D > charset=3D3Dus-ascii"><meta name=3D3DGenerator > content=3D3D&quot= > ;Microsoft Word 15 =3D > (filtered medium)"><style></style></head><body > l= > ang=3D3DES =3D > link=3D3D"#0563C1" vlink=3D3D"#954F72"><div > class= > =3D3DWordSection1> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Hi =3D > Ceph&#8217;s!<o:p></o:p> =3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>We started > det= > eling a =3D > bucket several days ago. Total size 47TB / 8.5M =3D > objects.<o:p></o:p> class=3D3DMsoNo= > rmal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Now we see > the= > cli =3D > bucket rm stucked and by console drop this =3D > messages.<o:p></o:p> class=3D3DMsoN= > ormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>[root@ceph-rgw > = > 03 ~]# =3D > 2019-10-28 13:55:43.880 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts = > =3D > WARNING : aborted 1000 incomplete multipart =3D > uploads<o:p></o:p> class=3D3DMsoNor= > mal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 56:24.021 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 2000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 57:04.726 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 3000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 57:45.424 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 4000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 58:25.905 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 5000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 59:06.898 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 6000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 59:47.829 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 7000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 00:42.102 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 8000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 01:23.829 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 9000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 02:06.028 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 10000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 02:48.648 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 11000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 03:29.807 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 12000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 04:11.180 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 13000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 04:52.396 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 14000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 05:33.050 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 15000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 06:13.652 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 16000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 06:54.806 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 17000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 07:35.867 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 18000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 08:16.886 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 19000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 08:57.711 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 20000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 09:38.032 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 21000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 10:18.377 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 22000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 10:58.833 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 23000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 11:39.078 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 24000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 12:24.731 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 25000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 13:12.176 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 26000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Bucket stats > s= > how 500K =3D > objects left. Looks like bucket rm is trying to abort all incompleted > =3D<b= > r> > mutipart. But in bucket stats this operation is not reflected removing > =3D<= > br> > objects from stats.<o:p></o:p> clas= > s=3D3DMsoNormal> > lang=3D3DEN-US =3D > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>May be wait > to= > get up =3D > 500K or it&#8217;s a > bug?<o:p></o:p>= > > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Regards<o:p= > ></o:p>=3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Manuel<o:p&= > gt;</o:p><=3D > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p></div></body></html> > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70-- > > ------------------------------ > > Date: Mon, 28 Oct 2019 10:48:44 -0400 > From: Casey Bodley <<a href=3D"mailto:cbodley@redhat.com" > target=3D"_bla= > nk">cbodley(a)redhat.com</a>> > Subject: [ceph-users] Re: Static website hosting with RGW > To: <a href=3D"mailto:ceph-users@ceph.io" > target=3D"_blank">ceph-users@ceph= > .io</a> > Message-ID: <<a href=3D"mailto: > 20834361-445e-1ee5-433b-dd4792f90608@redh= > at.com" target=3D"_blank">20834361-445e-1ee5-433b-dd4792f90608(a)redhat.com > </= > a>> > Content-Type: text/plain; charset=3DUTF-8; format=3Dflowed > > > On 10/24/19 8:38 PM, Oliver Freyermuth wrote: > > Dear Cephers, > > > > I have a question concerning static websites with RGW. > > To my understanding, it is best to run >=3D1 RGW client for > "c= > lassic" S3 and in addition operate >=3D1 RGW client for website > ser= > ving > > (potentially with HAProxy or its friends in front) to prevent messup > o= > f requests via the different protocols. > > > > I'd prefer to avoid "*.<a href=3D"http://example.com" > rel=3D"= > noreferrer" target=3D"_blank">example.com</a>" entries in DNS if > possi= > ble. > > So my current setup has these settings for the "web" RGW > cli= > ent: > >=C2=A0 =C2=A0rgw_enable_static_website =3D true > >=C2=A0 =C2=A0rgw_enable_apis =3D s3website > >=C2=A0 =C2=A0rgw_dns_s3website_name =3D > some_value_unused_when_A_record= > s_are_used_pointing_to_the_IP_but_it_needs_to_be_set > > and I create simple A records for each website pointing to the IP of > t= > his "web" RGW node. > > > > I can easily upload content for those websites to the other RGW > instan= > ces which are serving S3, > > so S3 and s3website APIs are cleanly separated in separate > instances.<= > br> > > > > However, one issue remains: How do I run > >=C2=A0 =C2=A0s3cmd ws-create > > on each website

1 0

Help
by Sumit Gaur 28 Oct '19

28 Oct '19

On Tue, 29 Oct 2019 at 1:50 am, <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: subtrees have overcommitted (target_size_bytes / > target_size_ratio) > (Lars Täuber) > 2. After delete 8.5M Objects in a bucket still 500K left > (EDH - Manuel Rios Fernandez) > 3. Re: Static website hosting with RGW (Casey Bodley) > > > ---------------------------------------------------------------------- > > Date: Mon, 28 Oct 2019 11:24:54 +0100 > From: Lars Täuber <taeuber(a)bbaw.de> > Subject: [ceph-users] Re: subtrees have overcommitted > (target_size_bytes / target_size_ratio) > To: ceph-users <ceph-users(a)ceph.io> > Message-ID: <20191028112454.0362fe66(a)bbaw.de> > Content-Type: text/plain; charset=UTF-8 > > Is there a way to get rid of this warnings with activated autoscaler > besides adding new osds? > > Yet I couldn't get a satisfactory answer to the question why this all > happens. > > ceph osd pool autoscale-status : > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET > RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > cephfs_data 122.2T 1.5 165.4T 1.1085 > 0.8500 1.0 1024 on > > versus > > ceph df : > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95 > > POOLS: > POOL ID STORED OBJECTS USED %USED > MAX AVAIL > cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 > 12 TiB > > > It seems that the overcommitment is wrongly calculated. Isn't the RATE > already used to calculate the SIZE? > > It seems USED(df) = SIZE(autoscale-status) > Isn't the RATE already taken into account here? > > Could someone please explain the numbers to me? > > > Thanks! > Lars > > Fri, 25 Oct 2019 07:42:58 +0200 > Lars Täuber <taeuber(a)bbaw.de> ==> Nathan Fish <lordcirth(a)gmail.com> : > > Hi Nathan, > > > > Thu, 24 Oct 2019 10:59:55 -0400 > > Nathan Fish <lordcirth(a)gmail.com> ==> Lars Täuber <taeuber(a)bbaw.de> : > > > Ah, I see! The BIAS reflects the number of placement groups it should > > > create. Since cephfs metadata pools are usually very small, but have > > > many objects and high IO, the autoscaler gives them 4x the number of > > > placement groups that it would normally give for that amount of data. > > > > > ah ok, I understand. > > > > > So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to > > > 0.3? Are the two pools using entirely different device classes, so > > > they are not sharing space? > > > > Yes, the metadata is on SSDs and the data on HDDs. > > > > > Anyway, I see that your overcommit is only "1.031x". So if you set > > > cephfs_data to 0.85, it should go away. > > > > This is not the case. I set the target_ratio to 0.7 and get this: > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > cephfs_metadata 15736M 3.0 2454G 0.0188 > 0.3000 4.0 256 on > > cephfs_data 122.2T 1.5 165.4T 1.1085 > 0.7000 1.0 1024 on > > > > The ratio seems to have nothing to do with the target_ratio but the SIZE > and the RAW_CAPACITY. > > Because the pool is still getting more data the SIZE increases and > therefore the RATIO increases. > > The RATIO seems to be calculated by this formula > > RATIO = SIZE * RATE / RAW_CAPACITY. > > > > This is what I don't understand. The data in the cephfs_data pool seems > to need more space than the raw capacity of the cluster provides. Hence the > situation is called "overcommitment". > > > > But why is this only the case when the autoscaler is active? > > > > Thanks > > Lars > > > > > > > > On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber(a)bbaw.de> > wrote: > > > > > > > > Thanks Nathan for your answer, > > > > > > > > but I set the the Target Ratio to 0.9. It is the cephfs_data pool > that makes the troubles. > > > > > > > > The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is > not explained on the page linked below. So I don't know its meaning. > > > > > > > > How can be a pool overcommited when it is the only pool on a set of > OSDs? > > > > > > > > Best regards, > > > > Lars > > > > > > > > Thu, 24 Oct 2019 09:39:51 -0400 > > > > Nathan Fish <lordcirth(a)gmail.com> ==> Lars Täuber <taeuber(a)bbaw.de> > : > > > > > The formatting is mangled on my phone, but if I am reading it > correctly, > > > > > you have set Target Ratio to 4.0. This means you have told the > balancer > > > > > that this pool will occupy 4x the space of your whole cluster, and > to > > > > > optimize accordingly. This is naturally a problem. Setting it to 0 > will > > > > > clear the setting and allow the autobalancer to work. > > > > > > > > > > On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber(a)bbaw.de> > wrote: > > > > > > > > > > > This question is answered here: > > > > > > https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ > > > > > > > > > > > > But it tells me that there is more data stored in the pool than > the raw > > > > > > capacity provides (taking the replication factor RATE into > account) hence > > > > > > the RATIO being above 1.0 . > > > > > > > > > > > > How comes this is the case? - Data is stored outside of the pool? > > > > > > How comes this is only the case when the autoscaler is active? > > > > > > > > > > > > Thanks > > > > > > Lars > > > > > > > > > > > > > > > > > > Thu, 24 Oct 2019 10:36:52 +0200 > > > > > > Lars Täuber <taeuber(a)bbaw.de> ==> ceph-users(a)ceph.io : > > > > > > > My question requires too complex an answer. > > > > > > > So let me ask a simple question: > > > > > > > > > > > > > > What does the SIZE of "osd pool autoscale-status" > tell/mean/comes from? > > > > > > > > > > > > > > Thanks > > > > > > > Lars > > > > > > > > > > > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > > > > > > Lars Täuber <taeuber(a)bbaw.de> ==> ceph-users(a)ceph.io : > > > > > > > > Hello everybody! > > > > > > > > > > > > > > > > What does this mean? > > > > > > > > > > > > > > > > health: HEALTH_WARN > > > > > > > > 1 subtrees have overcommitted pool > target_size_bytes > > > > > > > > 1 subtrees have overcommitted pool > target_size_ratio > > > > > > > > > > > > > > > > and what does it have to do with the autoscaler? > > > > > > > > When I deactivate the autoscaler the warning goes away. > > > > > > > > > > > > > > > > > > > > > > > > $ ceph osd pool autoscale-status > > > > > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY > RATIO > > > > > > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > > > > > > cephfs_metadata 15106M 3.0 2454G > 0.0180 > > > > > > 0.3000 4.0 256 on > > > > > > > > cephfs_data 113.6T 1.5 165.4T > 1.0306 > > > > > > 0.9000 1.0 512 on > > > > > > > > > > > > > > > > > > > > > > > > $ ceph health detail > > > > > > > > HEALTH_WARN 1 subtrees have overcommitted pool > target_size_bytes; 1 > > > > > > subtrees have overcommitted pool target_size_ratio > > > > > > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have > overcommitted > > > > > > pool target_size_bytes > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > 1.031x due > > > > > > to target_size_bytes 0 on pools [] > > > > > > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have > overcommitted > > > > > > pool target_size_ratio > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > 1.031x due > > > > > > to target_size_ratio 0.900 on pools ['cephfs_data'] > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > Lars > > > > > > > > _______________________________________________ > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 28 Oct 2019 14:18:01 +0100 > From: "EDH - Manuel Rios Fernandez" <mriosfer(a)easydatahost.com> > Subject: [ceph-users] After delete 8.5M Objects in a bucket still 500K > left > To: <ceph-users(a)ceph.io> > Message-ID: <02a201d58d92$1fe85880$5fb90980$(a)easydatahost.com> > Content-Type: multipart/alternative; > boundary="----=_NextPart_000_02A3_01D58D9A.81B17B70" > > This is a multipart message in MIME format. > > ------=_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/plain; > charset="us-ascii" > Content-Transfer-Encoding: 7bit > > Hi Ceph's! > > > > We started deteling a bucket several days ago. Total size 47TB / 8.5M > objects. > > > > Now we see the cli bucket rm stucked and by console drop this messages. > > > > [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700 0 > abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart uploads > > 2019-10-28 13:56:24.021 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 2000 incomplete multipart uploads > > 2019-10-28 13:57:04.726 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 3000 incomplete multipart uploads > > 2019-10-28 13:57:45.424 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 4000 incomplete multipart uploads > > 2019-10-28 13:58:25.905 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 5000 incomplete multipart uploads > > 2019-10-28 13:59:06.898 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 6000 incomplete multipart uploads > > 2019-10-28 13:59:47.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 7000 incomplete multipart uploads > > 2019-10-28 14:00:42.102 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 8000 incomplete multipart uploads > > 2019-10-28 14:01:23.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 9000 incomplete multipart uploads > > 2019-10-28 14:02:06.028 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 10000 incomplete multipart uploads > > 2019-10-28 14:02:48.648 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 11000 incomplete multipart uploads > > 2019-10-28 14:03:29.807 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 12000 incomplete multipart uploads > > 2019-10-28 14:04:11.180 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 13000 incomplete multipart uploads > > 2019-10-28 14:04:52.396 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 14000 incomplete multipart uploads > > 2019-10-28 14:05:33.050 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 15000 incomplete multipart uploads > > 2019-10-28 14:06:13.652 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 16000 incomplete multipart uploads > > 2019-10-28 14:06:54.806 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 17000 incomplete multipart uploads > > 2019-10-28 14:07:35.867 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 18000 incomplete multipart uploads > > 2019-10-28 14:08:16.886 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 19000 incomplete multipart uploads > > 2019-10-28 14:08:57.711 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 20000 incomplete multipart uploads > > 2019-10-28 14:09:38.032 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 21000 incomplete multipart uploads > > 2019-10-28 14:10:18.377 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 22000 incomplete multipart uploads > > 2019-10-28 14:10:58.833 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 23000 incomplete multipart uploads > > 2019-10-28 14:11:39.078 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 24000 incomplete multipart uploads > > 2019-10-28 14:12:24.731 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 25000 incomplete multipart uploads > > 2019-10-28 14:13:12.176 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 26000 incomplete multipart uploads > > > > > > Bucket stats show 500K objects left. Looks like bucket rm is trying to > abort > all incompleted mutipart. But in bucket stats this operation is not > reflected removing objects from stats. > > > > May be wait to get up 500K or it's a bug? > > > > Regards > > Manuel > > > > > ------=_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/html; > charset="us-ascii" > Content-Transfer-Encoding: quoted-printable > > <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = > xmlns:o=3D"urn:schemas-microsoft-com:office:office" = > xmlns:w=3D"urn:schemas-microsoft-com:office:word" = > xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = > xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = > http-equiv=3DContent-Type content=3D"text/html; = > charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 = > (filtered medium)"><style></style></head><body lang=3DES = > link=3D"#0563C1" vlink=3D"#954F72"><div class=3DWordSection1> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>Hi = > Ceph’s!<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>We started deteling a = > bucket several days ago. Total size 47TB / 8.5M = > objects.<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>Now we see the cli = > bucket rm stucked and by console drop this = > messages.<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>[root@ceph-rgw03 ~]# = > 2019-10-28 13:55:43.880 7f0dd92c9700  0 abort_bucket_multiparts = > WARNING : aborted 1000 incomplete multipart = > uploads<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:56:24.021 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 2000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:57:04.726 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 3000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:57:45.424 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 4000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:58:25.905 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 5000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:59:06.898 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 6000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:59:47.829 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 7000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:00:42.102 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 8000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:01:23.829 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 9000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:02:06.028 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 10000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:02:48.648 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 11000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:03:29.807 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 12000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:04:11.180 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 13000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:04:52.396 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 14000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:05:33.050 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 15000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:06:13.652 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 16000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:06:54.806 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 17000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:07:35.867 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 18000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:08:16.886 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 19000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:08:57.711 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 20000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:09:38.032 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 21000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:10:18.377 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 22000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:10:58.833 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 23000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:11:39.078 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 24000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:12:24.731 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 25000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:13:12.176 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 26000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>Bucket stats show 500K = > objects left. Looks like bucket rm is trying to abort all incompleted = > mutipart. But in bucket stats this operation is not reflected removing = > objects from stats.<o:p></o:p> lang=3DEN-US = > style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>May be wait to get up = > 500K or it’s a bug?<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>Regards<o:p></o:p>= > style=3D'color:#1F497D;mso-fareast-language:ES'>Manuel<o:p></o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p></div></body></html> > ------=_NextPart_000_02A3_01D58D9A.81B17B70-- > > ------------------------------ > > Date: Mon, 28 Oct 2019 10:48:44 -0400 > From: Casey Bodley <cbodley(a)redhat.com> > Subject: [ceph-users] Re: Static website hosting with RGW > To: ceph-users(a)ceph.io > Message-ID: <20834361-445e-1ee5-433b-dd4792f90608(a)redhat.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > > On 10/24/19 8:38 PM, Oliver Freyermuth wrote: > > Dear Cephers, > > > > I have a question concerning static websites with RGW. > > To my understanding, it is best to run >=1 RGW client for "classic" S3 > and in addition operate >=1 RGW client for website serving > > (potentially with HAProxy or its friends in front) to prevent messup of > requests via the different protocols. > > > > I'd prefer to avoid "*.example.com" entries in DNS if possible. > > So my current setup has these settings for the "web" RGW client: > > rgw_enable_static_website = true > > rgw_enable_apis = s3website > > rgw_dns_s3website_name = > some_value_unused_when_A_records_are_used_pointing_to_the_IP_but_it_needs_to_be_set > > and I create simple A records for each website pointing to the IP of > this "web" RGW node. > > > > I can easily upload content for those websites to the other RGW > instances which are serving S3, > > so S3 and s3website APIs are cleanly separated in separate instances. > > > > However, one issue remains: How do I run > > s3cmd ws-create > > on each website-bucket once? > > I can't do that against the "classic" S3-serving RGW nodes. This will > give me a 405 (not allowed), > > since they do not have rgw_enable_static_website enabled. > > I also can not run it against the "web S3" nodes, since they do not have > the S3 API enabled. > > Of course I could enable that, but then the RGW node can't cleanly > disentangle S3 and website requests since I use A records. > > > > Does somebody have a good idea on how to solve this issue? > > Setting "rgw_enable_static_website = true" on the S3-serving RGW nodes > would solve it, but does that have any bad side-effects on their S3 > operation? > > Enabling static website on the gateway serving the S3 api does look like > the right solution. As far as I can tell, it's only used to control > whether the S3 ops for PutBucketWebsite, GetBucketWebsite, and > DeleteBucketWebsite are exposed. > > > > > Also, if there's an expert on this: Exposing a bucket under a tenant as > static website is not possible since the colon (:) can't be encoded in DNS, > right? > > > > > > In case somebody also wants to set something like this up, here are the > best docs I could find: > > https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c > > and of course: > > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html-s… > > > > > > Cheers, > > Oliver > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 81, Issue 79 > ****************************************** >

1 0

Ceph monitor start error: monitor data filesystem reached concerning levels of available storage space
by Thomas Schneider 28 Oct '19

28 Oct '19

Hi, I'm facing an issue with starting 1 monitor. In the relevant error log the issue seems to be clear: root@ld5506:~# tail -n 30 /var/log/ceph/ceph-mon.ld5506.log 2019-10-28 17:14:46.471 7f157b097440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:14:46.471 7f157b097440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 421378 2019-10-28 17:14:46.471 7f157b097440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:14:56.563 7fb006b4c440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:14:56.563 7fb006b4c440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 421600 2019-10-28 17:14:56.563 7fb006b4c440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:15:06.823 7fb2c7084440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:15:06.823 7fb2c7084440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 421824 2019-10-28 17:15:06.823 7fb2c7084440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:15:17.067 7f722b0c4440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:15:17.067 7f722b0c4440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 422009 2019-10-28 17:15:17.067 7f722b0c4440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:15:27.323 7f34e37df440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:15:27.323 7f34e37df440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 422197 2019-10-28 17:15:27.323 7f34e37df440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) However there is now sufficient disk space available on this node after completing some file cleanup: root@ld5506:~# btrfs filesystem usage / Overall: Device size: 192.54GiB Device allocated: 183.54GiB Device unallocated: 9.00GiB Device missing: 0.00B Used: 110.76GiB Free (estimated): 75.01GiB (min: 70.51GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 237.86MiB (used: 0.00B) Data,single: Size:175.53GiB, Used:109.51GiB /dev/sdbq3 175.53GiB Metadata,DUP: Size:4.00GiB, Used:639.77MiB /dev/sdbq3 8.00GiB System,DUP: Size:8.00MiB, Used:48.00KiB /dev/sdbq3 16.00MiB Unallocated: /dev/sdbq3 9.00GiB I would expect that +70GB disk space must be sufficient for MON to start. Please advise how to fix this error. THX

1 0

Strage RBD images created
by Randall Smith 28 Oct '19

28 Oct '19

Greetings, I am running a mimic cluster. I noticed that I suddenly have over 200 rbd images that have seemingly random 64 character names. They were all created within a short time period. rbd info on one of the rbds looks like this: rbd image 'ff8f0ad0b5323fc3a8e930d96d3021d4c4dfc18c62ef9a1305b49456b8560904': size 16 GiB in 4096 objects order 22 (4 MiB objects) id: cb09e16b8b4567 block_name_prefix: rbd_data.cb09e16b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Fri Mar 1 11:19:14 2019 `rbd lock ls` shows that these images are not being used. I can certainly delete them but I found it odd that they showed up at all. Has anyone seen anything like this? -- Randall Smith Computing Services Adams State University http://www.adams.edu/ 719-587-7741

2 2

radosgw recovering shards
by Frank R 28 Oct '19

28 Oct '19

Hi all, Apologies for all the messages to the list over the past few days. After an upgrade from 12.2.7 to 12.2.12 (inherited cluster) for an RGW multisite active/active setup I am almost constantly seeing 1-10 "recovering shards" when running "radosgw-admin sync status", ie: ---------- # radosgw-admin sync status realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod) zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod) zone 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2) metadata sync no sync (zone is master) data sync source: ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shards behind shards: [102] 8 shards are recovering recovering shards: [31,37,56,60,83,92,95,127] ---------- Every once in a while it will go to full sync. This is seen on both the master and secondary side. There were a bunch of stale reshard instances (on both ends) that I was able to remove with: ----- radosgw-admin reshard stale-instances rm ----- What exactly is a "recovering shard". What can be performed to troubleshoot/fix this condition? I have verified that rgw_num_rados_handles is 1. Additionally, what exactly do: # radosgw-admin metadata sync init # radosgw-admin data sync init do and do they tend to take a long time to perform. thx Frank

1 0

RMDA Bug?
by Mason-Williams, Gabryel (DLSLtd,RAL,LSCI) 28 Oct '19

28 Oct '19

I am currently trying to run Ceph on RDMA, either RoCE 1 or 2. However, I am experiencing issues with this. When using Ceph on RDMA I experience issues where OSD’s will randomly become unreachable even if the cluster is left alone alone, it also is not properly talking over RDMA and using Ethernet when the config states it should as shown by the same results in the bench marking of the two setups. After reloading the cluster [cid:36020940-0085-40fc-bb5b-d91de6ace453] After 5m 9s the cluster went from being healthy to down. [cid:ed084bcc-0b97-44bd-9648-ce2e06859cd5] This problem even happens when running a bench mark test on the cluster, OSD’s will just fall over. Another curious issue is that it is not properly talking over RDMA as well and instead using the Ethernet. [cid:05e9dc68-075e-425d-b76b-ce7fa1d2f7a8] Next test: [cid:4183557e-b1da-41f3-afc3-f081b9fb4034] The config used for the RDMA is a so: [global] fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa mon_initial_members = node1, node2, node3 mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx auth_cluster_required = cephx auth_service_required =cephx auth_client_required = cephx public_network = xxx.xxx.xxx.xxx/24 cluster_network = yyy.yyy.yyy.yyy/16 ms_cluster_type =async+rdma ms_public_type = async+posix ms_async_rdma_device_name = mlx4_0 [osd.0] ms_async_rdma_local_gid = xxxx [osd.1] ms_async_rdma_local_gid = xxxx [osd.2] ms_async_rdma_local_gid =xxxx Tests to check the system is using RDMA sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep ms_cluster OUTPUT "ms_cluster_type": "async+rdma", sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1 OUTPUT { "AsyncMessenger::RDMAWorker-1": { "tx_no_mem": 0, "tx_parital_mem": 0, "tx_failed_post": 0, "rx_no_registered_mem": 0, "tx_chunks": 9, "tx_bytes": 2529, "rx_chunks": 0, "rx_bytes": 0, "pending_sent_conns": 0 } } When running over Ethernet I have a completely stable system with the current benchmarks as so [cid:544ecbbc-10d9-43e6-ab2f-aa7c2bcd88c0] Config setup when using Ethernet is The Config setup when using Ethernet is [global] fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa mon_initial_members = node1, node2, node3 mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx auth_cluster_required = cephx auth_service_required =cephx auth_client_required = cephx public_network = xxx.xxx.xxx.xxx/24 cluster_network = yyy.yyy.yyy.yyy/16 ms_cluster_type =async+posix ms_public_type = async+posix ms_async_rdma_device_name = mlx4_0 [osd.0] ms_async_rdma_local_gid = xxxx [osd.1] ms_async_rdma_local_gid = xxxx [osd.2] ms_async_rdma_local_gid =xxxx Tests to check the system is using async+posix sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep ms_cluster OUTPUT "ms_cluster_type": "async+posix" sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1 OUTPUT {} This clearly a issue with RDMA and not with the OSD's shown by the fact the system is completely fine over Ethernet and not with RDMA. Any guidance or ideas on how to approach this problem to make Ceph work with RDMA would be greatly appreciated. Regards Gabryel Mason-Williams, Placement Student Address: Diamond Light Source Ltd., Diamond House, Harwell Science & Innovation Campus, Didcot, Oxfordshire OX11 0DE Email: gabryel.mason-williams(a)diamond.ac.uk -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

Issue with cephfs
by LeeQ ＠ BitBahn.io 26 Oct '19

26 Oct '19

We have been using a cephfs pool to store machine data to, the data is not overly critical at this time but. Its got to around 8TB and we started to see kernel panics with the hosts that had the mounts in place. Now when try to start the MDS's they cycle through, Active, Replay, ClientReplay about 10 times and then just fail in a active(laggy)state. So I delete the MDS's (docker-croit)@us-croit-enc-deploy01 ~ $ ceph fs dump dumped fsmap epoch 5307 e5307 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 5307 flags 12 created 2019-10-26 20:43:02.087584 modified 2019-10-26 21:35:17.285598 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client -1 (unspecified) last_failure 0 last_failure_osd_epoch 2122066 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=267576193} failed damaged stopped data_pools [5,14] metadata_pool 3 inline_data disabled balancer standby_count_wanted 1 267576193: v1:100.129.255.186:6800/1355970155 'us-ceph-enc-svc02' mds.0.5301 up:active seq 16 laggy since 2019-10-26 21:12:08.027863 Looks ok. Then run (docker-croit)@us-croit-enc-deploy01 ~ $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data us_enc_datarefuge_001a ] (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-journal-tool --rank cephfs:all journal export lee.bak journal is 523855986688~99768 wrote 99768 bytes at offset 523855986688 to lee.bak NOTE: this is a _sparse_ file; you can $ tar cSzf lee.bak.tgz lee.bak to efficiently compress it while preserving sparseness. (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-journal-tool --rank cephfs:all event recover_dentries summary Events by type: RESETJOURNAL: 1 SESSION: 363 SESSIONS: 17 UPDATE: 14 Errors: 0 (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-journal-tool --rank cephfs:all journal reset old journal was 523855986688~99768 new journal start will be 523860180992 (4094536 bytes past old end) writing journal head writing EResetJournal entry done (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-table-tool all reset session { "0": { "data": {}, "result": 0 } } (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-table-tool all reset snap { "result": 0 } (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-table-tool all reset inode { "0": { "data": {}, "result": 0 } } Re add the MDS's and we go back round in a circle. Am I missing something? do I need to drop the metadata and recreate it maybe? If it comes to it I can drop all the data and start over, but don't really want to.

1 0

ceph pg commands hang forever
by Frank R 26 Oct '19

26 Oct '19

Hi all, I have an issue where all "ceph pg" command hang forever, ie: ceph pg dump ceph pg ls-by-pool etc All pgs report as active+clean. Any pointers on correcting this? thx Frank

1 1

0B OSDs
by DHilsbos＠performair.com 25 Oct '19

25 Oct '19

All; We're setting up our second cluster, using version 14.2.4, and we've run into a weird issue: all of our OSDs are created with a size of 0 B. Weights are appropriate for the size of the underlying drives, but ceph -s shows this: cluster: id: <id> health: HEALTH_WARN Reduced data availability: 256 pgs inactive too few PGs per OSD (28 < min 30) services: mon: 3 daemons, quorum s700041,s700042,s700043 (age 4d) mgr: s700041(active, since 3d), standbys: s700042, s700043 osd: 9 osds: 9 up (since 21m), 9 in (since 44m) data: pools: 1 pools, 256 pgs objects: 0 objects, 0 B -->usage: 0 B used, 0 B / 0 B avail<-- (emphasis added) pgs: 100.000% pgs unknown 256 unknown Thoughts? I have ceph-volumne.log, and the log from one of the OSD daemons, though it looks like the auth keys get printed to the ceph-volume.log. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com 300 S. Hamilton Pl. Gilbert, AZ 85233 Phone: (480) 610-3500 Fax: (480) 610-3501 www.PerformAir.com

2 1