Hi all,
In reference to:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2KTC7RXQYW…
We are seeing similar behavior with public Swift bucket access being broken.
In this case RadosGW Nautilus integrated to OpenStack Queens Keystone.
Public Swift containers have worked fine from Luminous era up to Nautilus
14.2.11, and started to break when upgrading RadosGW to 14.2.12 or newer.
Unsure if this is related to the backport of "rgw: Swift API anonymous access
should 401 (pr#37438", or some other rgw change within 14.2.12.
I believe the following ceph.conf we use is relevant:
rgw_swift_account_in_url = true
rgw_keystone_implicit_tenants = false
As well as the configured endpoint format:
https://fqdn:443/swift/v1/AUTH_%(tenant_id)s
Steps to reproduce:
Horizon:
--------
1) Public container access
- Create a container with "Container Access" set to Public
- Click on the Horizon provided Link which is of the format https://fqdn/swift/v1/AUTH_projectUUID/public-test-container/
Expected result: Empty bucket listing
Actual result: "AccessDenied"
2) Public object access
- Upload an object to the public container
- Try to access the object via unauthenticated browser session
Expected result: Object downloaded or loaded into browser
Actual result: "NoSuchBucket"
Also getting similar behavior with Swift CLI tools (ACL '.r:*') from what I
can see.
Any suggestions how to troubleshoot further?
Happy to provide more debug log and configuration details if need be, as well
as pointers if something might be actually wrong in our configuration.
Also, apologies for the possible double post - I tried to first submit via the
hyperkitty web form but that post seems to have gone into a black hole.
BR,
Jukka
Hi.
Thank You, I will try this solution probably on Sunday. Will write results here.
Regards
Mateusz Skała
> Wiadomość napisana przez Amit Ghadge <amitg.b14(a)gmail.com> w dniu 24.11.2020, o godz. 11:12:
>
> Sorry for delay reply, I never tried on production but after revert the changes I can able to reshard again
> You get same bucket two entries from metadata,
> radosgw-admin metadata list bucket.instance | grep bucket
> You now the older bucket entry then update those two parameter first,
> radosgw-admin metadata get bucket.instance:bucket:<id> > bucket.json
> Set reshard_status to 0 and new_bucket_instance_id to ""
> Update bucket instance by, radosgw-admin metadata put bucket.instance:bucket:<id> < bucket.json
>
> On Sun, Nov 22, 2020 at 6:04 PM Mateusz Skała <mateusz.skala(a)gmail.com <mailto:mateusz.skala@gmail.com>> wrote:
> Thank You for response, how I can upload this to metadata? Is this operation safe?
> Regards
> Mateusz Skała
>
> W dniu sob., 21.11.2020 o 18:01 Amit Ghadge <amitg.b14(a)gmail.com <mailto:amitg.b14@gmail.com>> napisał(a):
> I go through this and you need to update bucket metadata, radosgw-admin metadata get bucket.instance:bucket:xxx > bucket.json, update two parameter I don't remember but it's look reshard: false and next_marker set empty.
>
> -AmitG
> On Sat, 21 Nov, 2020, 2:04 PM Mateusz Skała, <mateusz.skala(a)gmail.com <mailto:mateusz.skala@gmail.com>> wrote:
> Hello Community.
> I need Your help. Few days ago I started manual resharding of one bucket with large objects. Unfortunately I interrupted this by Ctrl+c. At now I can’t start this process again.
> There is message:
> # radosgw-admin bucket reshard --bucket objects --num-shards 2
> ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time
>
> But list of reshard process is empty:
> # radosgw-admin reshard list
> []
>
> # radosgw-admin reshard status --bucket objects
> [
> {
> "reshard_status": "not-resharding",
> "new_bucket_instance_id": "",
> "num_shards": -1
> }
> ]
>
> How can I fix this situation ? How to restore possibility resharding this bucket?
> And BTW is resharding process locking writes/reads on bucket?
> Regards
> Mateusz Skała
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>
> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>
Hi all,
After I upgrade from 14.2.9 to 14.2.14 my OSDs are using less more memory
than before! I give each OSD 6GB memory target and before the free memory
was 20GB and now after 24h from the upgrade I have 104GB free memory of
128GB memory! Also, my OSD latency got increases!
This happens in both SSD and HDD tier.
Are there any notes from the upgrade I missed? Is it related to
bluefs_buffered_io?
If BlueFS do a direct IO shouldn't BlueFS/Bluestore use the targeted memory
for its cache and does it mean before the upgrade the memory used was by a
kernel that buffers the IO and wasn't for the ceph-osd?
Thanks.
Hi,
Depending on which cluster I look at (all running v14.2.11), the
bytes_used is reporting raw space or stored bytes variably.
Here's a 7 year old cluster:
# ceph df -f json | jq .pools[0]
{
"name": "volumes",
"id": 4,
"stats": {
"stored": 1229308190855881,
"objects": 294401604,
"kb_used": 1200496280133,
"bytes_used": 1229308190855881,
"percent_used": 0.4401889145374298,
"max_avail": 521125025021952
}
}
Note that stored == bytes_used for that pool. (this is a 3x replica pool).
But here's a newer cluster (installed recently with nautilus)
# ceph df -f json | jq .pools[0]
{
"name": "volumes",
"id": 1,
"stats": {
"stored": 680977600893041,
"objects": 163155803,
"kb_used": 1995736271829,
"bytes_used": 2043633942351985,
"percent_used": 0.23379847407341003,
"max_avail": 2232457428467712
}
}
In the second cluster, bytes_used is 3x stored.
Does anyone know why these are not reported consistently?
Noticing this just now, I'll update our monitoring to plot stored
rather than bytes_used from now on.
Thanks!
Dan
Hello,
We are planning to perform a small upgrade to our cluster and slowly start adding 12TB SATA HDD drives. We need to accommodate for additional SSD WAL/DB requirements as well. Currently we are considering the following:
HDD Drives - Seagate EXOS 12TB
SSD Drives for WAL/DB - Intel D3 S4510 960GB or Intel D3 S4610 960GB
Our cluster isn't hosting any IO intensive DBs nor IO hungry VMs such as Exchange, MSSQL, etc.
From the documentation that I've read the recommended size for DB is between 1% and 4% of the size of the osd. Would 2% figure be sufficient enough (so around 240GB DB size for each 12TB osd?)
Also, from your experience, which is a better model for the SSD DB/WAL? Would Intel S4510 be sufficient enough for our purpose or would the S4610 be a much better choice? Are there any other cost effective performance to consider instead of the above models?
The same question to the HDD. Any other drives we should consider instead of the Seagate EXOS series?
Thanks for you help and suggestions.
Andrei
Hi,
I am currently evaluating ceph and stumbled across an odd issue when an osd comes back online.
The osd was taken offline, but is still "in" and is brought back online before it is marked "out".
As a test I run a fio job with 4k rand I/O on a 10G rbd volume during the OSD down and up procedure.
As OSDs I use 8x 960GB SAS SSDs on 4 Nodes interconnected with 2x10GE each. Network and SSD seems not to be congested at any time.
From time to time I see complete stall in the fio benchmark for approx. 10 seconds while recovery is ongoing.
All recovery parameteres (max_recovery, max_backfill, sleep etc.) do not seem to influence it.
While digging deeper I found that the requests are hanging in the "waiting for readable" state.
Help how to debug this further would be great. Might it be linked to the new feature in Octopus to read from
all OSDs and not just the primary?
Thank you,
Peter
Hi,
I have k8s cephfs-provisioner.yaml and storageclass.yaml:
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cephfs
namespace: cephfs
provisioner: ceph.com/cephfs
parameters:
monitors: 10.32.121.51:6789,10.32.121.52:6789,10.32.121.53:6789
adminId: admin
adminSecretName: ceph-admin-secret
adminSecretNamespace: cephfs
claimRoot: /pvc-volumes
reclaimPolicy: Retain
allowVolumeExpansion: true
# ceph version
ceph version 15.2.5
I'm trying to create snapshots:
/source # cd .snap
/source/.snap # mkdir snapshot01
mkdir: can't create directory 'snapshot01': Permission denied
/source/.snap #
# ceph auth ls shows client permissions:
client.kubernetes-dynamic-user-a89595fe-2a80-11eb-8b50-1ec01fe0b788
key: AQAjl7Zf9mFbMxAAcGiyth980XbD1rtcBokqAw==
caps: [mds] allow r,allow rw
path=/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-a89595cc-2a80-11eb-8b50-1ec01fe0b788
caps: [mon] allow r
caps: [osd] allow rw pool=cephfs.cephfsvol1.data
namespace=fsvolumens_kubernetes-dynamic-pvc-a89595cc-2a80-11eb-8b50-1ec01fe0b788
If I understood correctly that above permission is lacking 's' flag.
How can I enable that permission when client is dynamic as above? Is
there a way to get around this problem, e.g. how can I define client
setup so that it wouldn't be dynamic and I could give required
permissions to that? I'm trying to have a backup Pod to generate
snapshots and copy those to nfs mount.
Br, Jouni
Hey Timothy,
Did you ever resolve this issue, and if so, how?
> Thank you..I looked through both logs and noticed this in the cancel one:
>
> osd_op(unknown.0.0:4164 41.2 41:55b0279d:reshard::reshard.0000000009:head [call
> rgw.reshard_remove] snapc 0=[] ondisk+write+known_if_redirected e24984) v8 --
> 0x7fe9b3625710 con 0
> osd_op_reply(4164 reshard.0000000009 [call] v24984'105796943 uv105796922 ondisk = -2
> ((2) No such file or directory)) v8 ==== 162+0+0 (203651653 0 0) 0x7fe9880044a0 con
> 0x7fe9b3625b70
> ERROR: failed to remove entry from reshard log, oid=reshard.0000000009 tenant= bucket=foo
>
> Is there anything else that I should look for? It looks like the cancel process thinks
> that reshard.0000000009 is present (and probably blocking my attempts at resharding) but
> it's not actually there and thus can't be removed.
Eric
--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA
Hey all,
We will be having a Ceph science/research/big cluster call on Wednesday
November 25th. If anyone wants to discuss something specific they can
add it to the pad linked below. If you have questions or comments you
can contact me.
This is an informal open call of community members mostly from
hpc/htc/research environments where we discuss whatever is on our minds
regarding ceph. Updates, outages, features, maintenance, etc...there is
no set presenter but I do attempt to keep the conversation lively.
https://pad.ceph.com/p/Ceph_Science_User_Group_20201125
<https://pad.ceph.com/p/Ceph_Science_User_Group_20201125>
<https://pad.ceph.com/p/Ceph_Science_User_Group_20200923>
We try to keep it to an hour or less.
Ceph calendar event details:
November 25th, 2020
15:00 UTC
4pm Central European
9am Central US
Description: Main pad for discussions:
https://pad.ceph.com/p/Ceph_Science_User_Group_Index
Meetings will be recorded and posted to the Ceph Youtube channel.
To join the meeting on a computer or mobile phone:
https://bluejeans.com/908675367?src=calendarLink
To join from a Red Hat Deskphone or Softphone, dial: 84336.
Connecting directly from a room system?
1.) Dial: 199.48.152.152 or bjn.vc
2.) Enter Meeting ID: 908675367
Just want to dial in on your phone?
1.) Dial one of the following numbers: 408-915-6466 (US)
See all numbers: https://www.redhat.com/en/conference-numbers
2.) Enter Meeting ID: 908675367
3.) Press #
Want to test your video connection? https://bluejeans.com/111
Kevin
--
Kevin Hrpcek
NASA VIIRS Atmosphere SIPS
Space Science & Engineering Center
University of Wisconsin-Madison
Forwarding here in case anyone is seeing the same/similar issue, Amit gave
really good pointers and a workaround :)
Thanks Amit!
On 11/25 16:08, Amit Ghadge wrote:
> Yes, and if you want avoid in future update this flag to 0 by $echo 0 >
> /sys/block/sdx/queue/rotational
>
> Thanks
>
> On Wed, Nov 25, 2020 at 4:03 PM David Caro <dcaro(a)wikimedia.org> wrote:
>
> >
> > Yep, you are right:
> >
> > ```
> > # cat /sys/block/sdd/queue/rotational
> > 1
> > ```
> >
> > I was looking to the code too but you got there before me :)
> >
> > https://github.com/ceph/ceph/blob/25ac1528419371686740412616145703810a561f/…
> >
> >
> > It might be an issue with the driver then reporting the wrong data. I'll
> > look
> > into it.
> >
> > Do you mind if I reply on the list with this info? (or if you want you
> > reply)
> > I think this might help others too (and myself in the future xd)
> >
> > Thanks Amit!
> >
> > On 11/25 15:50, Amit Ghadge wrote:
> > > This might happen when the disk default sets 1
> > > in /sys/block/sdx/queue/rotational , 1 for HDD and 0 for SSD, But we not
> > > see any problem till now.
> > >
> > > -AmitG
> > >
> > > On Wed, Nov 25, 2020 at 3:08 PM David Caro <dcaro(a)wikimedia.org> wrote:
> > >
> > > >
> > > > Hi!
> > > >
> > > > I have a nautilus ceph cluster, and today I restarted one of the osd
> > > > daemons
> > > > and spend some time trying to debug an error I was seeing in the log,
> > > > though it
> > > > seems the osd is actually working.
> > > >
> > > >
> > > > The error I was seeing is:
> > > > ```
> > > > Nov 25 09:07:43 osd15 systemd[1]: Starting Ceph object storage daemon
> > > > osd.44...
> > > > Nov 25 09:07:43 osd15 systemd[1]: Started Ceph object storage daemon
> > > > osd.44.
> > > > Nov 25 09:07:47 osd15 ceph-osd[12230]: 2020-11-25 09:07:47.846
> > > > 7f55395fbc80 -1 osd.44 106947 log_to_monitors {default=true}
> > > > Nov 25 09:07:47 osd15 ceph-osd[12230]: 2020-11-25 09:07:47.850
> > > > 7f55395fbc80 -1 osd.44 106947 mon_cmd_maybe_osd_create fail: 'osd.44
> > has
> > > > already bound to class 'ssd', can not reset class to 'hdd'; use 'ceph
> > osd
> > > > crush rm-device-class <id>' to remove old class first': (16) Device or
> > > > resource busy
> > > > ```
> > > >
> > > > There's no other messages in the journal so at first I thought that
> > the osd
> > > > failed to start.
> > > > But it seems to be up and working correctly anyhow.
> > > >
> > > > There's no "hdd" class in my crush map:
> > > > ```
> > > > # ceph osd crush class ls
> > > > [
> > > > "ssd"
> > > > ]
> > > > ```
> > > >
> > > > And that osd is actually of the correct class:
> > > > ```
> > > > # ceph osd crush get-device-class osd.44
> > > > ssd
> > > > ```
> > > >
> > > > ```
> > > > # uname -a
> > > > Linux osd15 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1
> > (2020-06-07)
> > > > x86_64 GNU/Linux
> > > >
> > > > # ceph --version
> > > > ceph version 14.2.5-1-g23e76c7aa6
> > > > (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)
> > > > ```
> > > >
> > > > The osd shows up in the cluster and it's receiving load, so there
> > seems to
> > > > be
> > > > no problem, but does anyone know what that error is about?
> > > >
> > > >
> > > > Thanks!
> > > >
> > > >
> > > > --
> > > > David Caro
> > > > SRE - Cloud Services
> > > > Wikimedia Foundation <https://wikimediafoundation.org/>
> > > > PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
> > > >
> > > > "Imagine a world in which every single human being can freely share in
> > the
> > > > sum of all knowledge. That's our commitment."
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > >
> >
> > --
> > David Caro
> > SRE - Cloud Services
> > Wikimedia Foundation <https://wikimediafoundation.org/>
> > PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
> >
> > "Imagine a world in which every single human being can freely share in the
> > sum of all knowledge. That's our commitment."
> >
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
