- ceph-users - lists.ceph.io

DB sizing for lots of large files
by Richard Thornton 28 Nov '20

28 Nov '20

Hi, Sorry to bother you all. It’s a home server setup. Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB 7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from eMMC). Only CephFS, I'm anticipating having 50-200K files when the 50TB (4+2 EC) is full. I'm trying to address the issue of really big OSD's, not my words, a Redditor: "When you write an object to a drive with collocated db and raw space, the disk has to read/write to both sections before acking a write. That's a lot to ask a 7200 disk to handle gracefully. I believe Red Hat only supports up to 8TB because of performance concerns with larger disks. I may be wrong, but once you are shuffling through 6-10TB of data I'd think those disks are gonna be bogged down in seek time." So I want to have my two DB's on the Optane to avoid the above, am I making sense? OK, so large files have lower metadata overhead than small files. This is for a media library this probably means super low overhead, one guy I spoke to had similar setup and for 48TB used he had a 2.6GB DB? Is there a rough CephFS calculation (each file uses x bytes of metadata), I think I should be safe with 30GB, now I read I should double that (you should allocate twice the size of the biggest layer to allow for compaction) but I only have 118GB and two OSDs so I will have to go for 59GB (or whatever will fit)? I'm thinking that I might not even use 3GB but to be safe I’ll make it 30GB, lets say it settles at 5GB, when compaction comes that means I will only need 20GB and therefore no spillover? I realise that if the Optane dies both OSD's go with it, do I have to configure anything special there (CRUSH would just handle it)? Being that this a home deployment will Ceph be OK with an occasional power outage, I mean if a bluray mkv gets corrupted ripping it again is an easy fix, also I will backup the clusters data once a month? Below are the commands I got from the website: Create the volume groups: $ vgcreate ceph-block-0 /dev/sda $ vgcreate ceph-block-1 /dev/sdb Create the logical volumes: $ lvcreate -l 100%FREE -n block-0 ceph-block-0 $ lvcreate -l 100%FREE -n block-1 ceph-block-1 Create db logical volumes (118GB Optane) $ vgcreate ceph-db-0 /dev/sdc $ lvcreate -L 59GB -n db-0 ceph-db-0 $ lvcreate -L 59GB -n db-1 ceph-db-0 Create the OSDs $ ceph-volume lvm create --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 $ ceph-volume lvm create --bluestore --data ceph-block-1/block-1 --block.db ceph-db-0/db-1 Thanks. Richard

5 5

Access/Delete RGW user with leading whitespace
by Benjamin.Zieglmeier 27 Nov '20

27 Nov '20

Hello, In our environment we have a user that has a leading whitespace in the UID. I don’t know how it was created, however I am unable to GET or DELETE it either using `radosgw-admin` or the Admin API: # radosgw-admin user list | grep rgw " rgw-prometheus", "rgw-prometheus", When I try to get info of the user directly, I get: # radosgw-admin user info --uid=" rgw-prometheus" could not fetch user info: no user info saved I’ve tried using the admin API as well while URL encoding the whitespace to %20 and get “Invalid Argument”. This user is not important in any way, however it creates issues trying to monitor the rgw usage logs using https://github.com/blemmenes/radosgw_usage_exporter I’ve considered modifying the script to ignore that user, but clearly Ceph is having troubles addressing it as well, so I figured I’d try to get to the bottom of how to remove this user. We are running 12.2.11 currently, however this cluster was built on 12.2.5 and I’m 99% certain the user was created in 12.2.5 Thanks, Ben

1 1

Tracing in ceph
by Seena Fallah 27 Nov '20

27 Nov '20

Hi all, Does this project work with the latest zipkin apis? https://github.com/ceph/babeltrace-zipkin Also what do you prefer to trace requests for rgw and rbd in ceph? Thanks.

2 3

Public Swift yielding errors since 14.2.12
by Jukka Nousiainen 27 Nov '20

27 Nov '20

Hi all, In reference to: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2KTC7RXQYW… We are seeing similar behavior with public Swift bucket access being broken. In this case RadosGW Nautilus integrated to OpenStack Queens Keystone. Public Swift containers have worked fine from Luminous era up to Nautilus 14.2.11, and started to break when upgrading RadosGW to 14.2.12 or newer. Unsure if this is related to the backport of "rgw: Swift API anonymous access should 401 (pr#37438", or some other rgw change within 14.2.12. I believe the following ceph.conf we use is relevant: rgw_swift_account_in_url = true rgw_keystone_implicit_tenants = false As well as the configured endpoint format: https://fqdn:443/swift/v1/AUTH_%(tenant_id)s Steps to reproduce: Horizon: -------- 1) Public container access - Create a container with "Container Access" set to Public - Click on the Horizon provided Link which is of the format https://fqdn/swift/v1/AUTH_projectUUID/public-test-container/ Expected result: Empty bucket listing Actual result: "AccessDenied" 2) Public object access - Upload an object to the public container - Try to access the object via unauthenticated browser session Expected result: Object downloaded or loaded into browser Actual result: "NoSuchBucket" Also getting similar behavior with Swift CLI tools (ACL '.r:*') from what I can see. Any suggestions how to troubleshoot further? Happy to provide more debug log and configuration details if need be, as well as pointers if something might be actually wrong in our configuration. Also, apologies for the possible double post - I tried to first submit via the hyperkitty web form but that post seems to have gone into a black hole. BR, Jukka

2 2

Re: Manual bucket resharding problem
by Mateusz Skała 27 Nov '20

27 Nov '20

Hi. Thank You, I will try this solution probably on Sunday. Will write results here. Regards Mateusz Skała > Wiadomość napisana przez Amit Ghadge <amitg.b14(a)gmail.com> w dniu 24.11.2020, o godz. 11:12: > > Sorry for delay reply, I never tried on production but after revert the changes I can able to reshard again > You get same bucket two entries from metadata, > radosgw-admin metadata list bucket.instance | grep bucket > You now the older bucket entry then update those two parameter first, > radosgw-admin metadata get bucket.instance:bucket:<id> > bucket.json > Set reshard_status to 0 and new_bucket_instance_id to "" > Update bucket instance by, radosgw-admin metadata put bucket.instance:bucket:<id> < bucket.json > > On Sun, Nov 22, 2020 at 6:04 PM Mateusz Skała <mateusz.skala(a)gmail.com <mailto:mateusz.skala@gmail.com>> wrote: > Thank You for response, how I can upload this to metadata? Is this operation safe? > Regards > Mateusz Skała > > W dniu sob., 21.11.2020 o 18:01 Amit Ghadge <amitg.b14(a)gmail.com <mailto:amitg.b14@gmail.com>> napisał(a): > I go through this and you need to update bucket metadata, radosgw-admin metadata get bucket.instance:bucket:xxx > bucket.json, update two parameter I don't remember but it's look reshard: false and next_marker set empty. > > -AmitG > On Sat, 21 Nov, 2020, 2:04 PM Mateusz Skała, <mateusz.skala(a)gmail.com <mailto:mateusz.skala@gmail.com>> wrote: > Hello Community. > I need Your help. Few days ago I started manual resharding of one bucket with large objects. Unfortunately I interrupted this by Ctrl+c. At now I can’t start this process again. > There is message: > # radosgw-admin bucket reshard --bucket objects --num-shards 2 > ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time > > But list of reshard process is empty: > # radosgw-admin reshard list > [] > > # radosgw-admin reshard status --bucket objects > [ > { > "reshard_status": "not-resharding", > "new_bucket_instance_id": "", > "num_shards": -1 > } > ] > > How can I fix this situation ? How to restore possibility resharding this bucket? > And BTW is resharding process locking writes/reads on bucket? > Regards > Mateusz Skała > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

OSD Memory usage
by Seena Fallah 26 Nov '20

26 Nov '20

Hi all, After I upgrade from 14.2.9 to 14.2.14 my OSDs are using less more memory than before! I give each OSD 6GB memory target and before the free memory was 20GB and now after 24h from the upgrade I have 104GB free memory of 128GB memory! Also, my OSD latency got increases! This happens in both SSD and HDD tier. Are there any notes from the upgrade I missed? Is it related to bluefs_buffered_io? If BlueFS do a direct IO shouldn't BlueFS/Bluestore use the targeted memory for its cache and does it mean before the upgrade the memory used was by a kernel that buffers the IO and wasn't for the ceph-osd? Thanks.

2 4

ceph df: pool stored vs bytes_used -- raw or not?
by Dan van der Ster 26 Nov '20

26 Nov '20

Hi, Depending on which cluster I look at (all running v14.2.11), the bytes_used is reporting raw space or stored bytes variably. Here's a 7 year old cluster: # ceph df -f json | jq .pools[0] { "name": "volumes", "id": 4, "stats": { "stored": 1229308190855881, "objects": 294401604, "kb_used": 1200496280133, "bytes_used": 1229308190855881, "percent_used": 0.4401889145374298, "max_avail": 521125025021952 } } Note that stored == bytes_used for that pool. (this is a 3x replica pool). But here's a newer cluster (installed recently with nautilus) # ceph df -f json | jq .pools[0] { "name": "volumes", "id": 1, "stats": { "stored": 680977600893041, "objects": 163155803, "kb_used": 1995736271829, "bytes_used": 2043633942351985, "percent_used": 0.23379847407341003, "max_avail": 2232457428467712 } } In the second cluster, bytes_used is 3x stored. Does anyone know why these are not reported consistently? Noticing this just now, I'll update our monitoring to plot stored rather than bytes_used from now on. Thanks! Dan

2 13

Advice on SSD choices for WAL/DB?
by Andrei Mikhailovsky 26 Nov '20

26 Nov '20

Hello, We are planning to perform a small upgrade to our cluster and slowly start adding 12TB SATA HDD drives. We need to accommodate for additional SSD WAL/DB requirements as well. Currently we are considering the following: HDD Drives - Seagate EXOS 12TB SSD Drives for WAL/DB - Intel D3 S4510 960GB or Intel D3 S4610 960GB Our cluster isn't hosting any IO intensive DBs nor IO hungry VMs such as Exchange, MSSQL, etc. From the documentation that I've read the recommended size for DB is between 1% and 4% of the size of the osd. Would 2% figure be sufficient enough (so around 240GB DB size for each 12TB osd?) Also, from your experience, which is a better model for the SSD DB/WAL? Would Intel S4510 be sufficient enough for our purpose or would the S4610 be a much better choice? Are there any other cost effective performance to consider instead of the above models? The same question to the HDD. Any other drives we should consider instead of the Seagate EXOS series? Thanks for you help and suggestions. Andrei

4 4

octupus: stall i/o during recovery
by Peter Lieven 26 Nov '20

26 Nov '20

Hi, I am currently evaluating ceph and stumbled across an odd issue when an osd comes back online. The osd was taken offline, but is still "in" and is brought back online before it is marked "out". As a test I run a fio job with 4k rand I/O on a 10G rbd volume during the OSD down and up procedure. As OSDs I use 8x 960GB SAS SSDs on 4 Nodes interconnected with 2x10GE each. Network and SSD seems not to be congested at any time. From time to time I see complete stall in the fio benchmark for approx. 10 seconds while recovery is ongoing. All recovery parameteres (max_recovery, max_backfill, sleep etc.) do not seem to influence it. While digging deeper I found that the requests are hanging in the "waiting for readable" state. Help how to debug this further would be great. Might it be linked to the new feature in Octopus to read from all OSDs and not just the primary? Thank you, Peter

1 0

snap permission denied
by vcjouni 26 Nov '20

26 Nov '20

Hi, I have k8s cephfs-provisioner.yaml and storageclass.yaml: --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: cephfs namespace: cephfs provisioner: ceph.com/cephfs parameters: monitors: 10.32.121.51:6789,10.32.121.52:6789,10.32.121.53:6789 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: cephfs claimRoot: /pvc-volumes reclaimPolicy: Retain allowVolumeExpansion: true # ceph version ceph version 15.2.5 I'm trying to create snapshots: /source # cd .snap /source/.snap # mkdir snapshot01 mkdir: can't create directory 'snapshot01': Permission denied /source/.snap # # ceph auth ls shows client permissions: client.kubernetes-dynamic-user-a89595fe-2a80-11eb-8b50-1ec01fe0b788 key: AQAjl7Zf9mFbMxAAcGiyth980XbD1rtcBokqAw== caps: [mds] allow r,allow rw path=/pvc-volumes/kubernetes/kubernetes-dynamic-pvc-a89595cc-2a80-11eb-8b50-1ec01fe0b788 caps: [mon] allow r caps: [osd] allow rw pool=cephfs.cephfsvol1.data namespace=fsvolumens_kubernetes-dynamic-pvc-a89595cc-2a80-11eb-8b50-1ec01fe0b788 If I understood correctly that above permission is lacking 's' flag. How can I enable that permission when client is dynamic as above? Is there a way to get around this problem, e.g. how can I define client setup so that it wouldn't be dynamic and I could give required permissions to that? I'm trying to have a backup Pod to generate snapshots and copy those to nfs mount. Br, Jouni

1 0