- ceph-users - lists.ceph.io

Erasure Profile Pool caps at pg_num 1024
by Gunnar Bandelow 17 Feb '20

17 Feb '20

Hello Everyone, i've run into problems with placement groups. We have a 12 Host Ceph-Cluster with 408 OSDs (hdd and ssd). I create a replicated pool with a large pg_num (16384). No problems everything works. If I do this with an erasure pool, i get a warning which is fixable by extending the max_pgs_per_osd and afterwards i get a health_warn in the ceph status due too few PGs. Checking the pg_num after pool creation, it is capped at 1024. I'm stuck at this point. Maybe i did something fundamentally wrong? To illustrate my steps I tried to summarize everything in a small example: # ceph -v ceph version 14.2.7 (fb8e34a687d76cd3bd45c2a0fb445432ab69b4ff) nautilus (stable) # ceph osd erasure-code-profile get myerasurehdd crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=7 m=5 plugin=jerasure technique=reed_sol_van w=8 # ceph osd crush rule dump sas_rule { "rule_id": 0, "rule_name": "sas_rule", "ruleset": 0, "type": 3, "min_size": 1, "max_size": 12, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] } # ceph osd pool create sas-pool 16384 16384 erasure myerasurehdd sas_rule Error ERANGE: pg_num 16384 size 12 would mean 196704 total pgs, which exceeds max 102000 (mon_max_pg_per_osd 250 * num_in_osds 408) # ceph tell mon.\* injectargs '--mon-max-pg-per-osd=500' mon.ceph-fs01: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) mon.ceph-fs05: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) mon.ceph-fs09: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) ceph-fs01:/opt/ceph-setup# ceph osd pool create sas-pool 16384 16384 erasure myerasurehdd sas_rule pool 'sas-pool' created # ceph -s cluster: id: b9471b57-95a2-4e58-8f69-b5e6048bea7c health: HEALTH_WARN Reduced data availability: 1024 pgs incomplete too few PGs per OSD (7 < min 30) # ceph osd pool get sas-pool pg_num pg_num: 1024 Best regards, Gunnar

3 2

Re: Bucket rename with
by Janne Johansson 17 Feb '20

17 Feb '20

Den fre 14 feb. 2020 kl 23:02 skrev EDH - Manuel Rios < mriosfer(a)easydatahost.com>: > Honestly, not having a function to rename bucket from admin rgw-admin is > like not having a function to copy or move. It is something basic, since if > not the workarround, it is to create a new bucket and move all the files > with the consequent loss of time and cost of computation. In addition to > the interruption. > Im sure that im not the only one administrator of rgw that need rename > some buckets, because by default system let Users for example use CAPITIAL > letter and that's not compliance. > I wonder if the "bucketnames might also become hostnames" weighed into the reluctance for renames, but I have also wanted to have renames, in the reverse situation where I did copy contents to another bucket (placed on a better place) and wanted to "copy OLD to NEW, rename OLD to OLD2, rename NEW to OLD, slowly delete OLD2" to have the shortest possible downtime while still doing a full object copy. -- May the most significant bit of your life be positive.

1 0

Extended security attributes on cephfs (nautilus) not working with kernel 5.3
by Stolte, Felix 17 Feb '20

17 Feb '20

Hi guys, I am exporting cephfs with samba using the vfs acl_xattr which stores ntfs acls in the security extended attributes. This works fine using cephfs kernel mount wither kernel version 4.15. Using kernel 5.3 I cannot access the security.ntacl attributes anymore. Attributes in user or ceph namespace is still working. Can someone enlighten me on this issue (or even better tell me how to fix it)? Best regards Felix ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

2 2

RBD-mirror instabilities
by Oliver Freyermuth 15 Feb '20

15 Feb '20

Dear Cephalopodians, running 13.2.6 on the source cluster and 14.2.5 on the rbd mirror nodes and the target cluster, I observe regular failures of rbd-mirror processes. With failures, I mean that traffic stops, but the daemons are still listed as active rbd-mirror daemons in "ceph -s", and the daemons are still running. This comes in sync with a hefty load of below messages in the mirror logs. This happens "sometimes" when some OSDs go down and up in the target cluster (which happens each night since the disks in that cluster shortly go offline during "online" smart self-tests - that's a problem in itself, but it's a cluster built from hardware that would have been trashed otherwise). The rbd daemons keep running in any case, but synchronization stops. If not all rbd mirror daemons have failed (we have three running, and it usually does not hit all of them), the "surviving" seem(s) not to take care of the images the other daemons had locked. Right now, I am eyeing with a "quick solution" of regularly restarting the rbd-mirror daemons, but if there are any good ideas on which debug info I could collect to get this analyzed and fixed, that would of course be appreciated :-). Cheers, Oliver ----------------------------------------------- 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_process_entry_safe: failed to commit journal event: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_replay_complete: replay encountered an error: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_process_entry_safe: failed to commit journal event: (125) Operation canceled 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_replay_complete: replay encountered an error: (125) Operation canceled 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dce2e05b0 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dcf47ee70 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:55.716 7f31f5b6f700 -1 rbd::mirror::ImageReplayer: 0x559dcb997680 [2/f8218221-6608-4a2b-8831-84ca0c2cb418] operator(): start failed: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0577bd16-acc4-4e9a-81f0-c698a24f8771: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=05bd4cca-a561-4a5c-ad83-9905ad5ce34e: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0e614ece-65b1-4b4a-99bd-44dd6235eb70: blacklisted detected during image replay -----------------------------------------------

2 6

Re: Bucket rename with
by Matt Benjamin 15 Feb '20

15 Feb '20

The world lived for a long time without it, but it's certainly useful. If Abhishek and/or the backport team would like this for Nautilus, I will help retarget our downstream backport (it's bigger than 22 commits with the dependencies, I believe, n.b.). Matt On Fri, Feb 14, 2020 at 5:02 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > Honestly, not having a function to rename bucket from admin rgw-admin is like not having a function to copy or move. It is something basic, since if not the workarround, it is to create a new bucket and move all the files with the consequent loss of time and cost of computation. In addition to the interruption. > > Im sure that im not the only one administrator of rgw that need rename some buckets, because by default system let Users for example use CAPITIAL letter and that's not compliance. > > KR, > Manuel > > -----Mensaje original----- > De: J. Eric Ivancich <ivancich(a)redhat.com> > Enviado el: viernes, 14 de febrero de 2020 20:47 > Para: EDH - Manuel Rios <mriosfer(a)easydatahost.com> > CC: ceph-users(a)ceph.io > Asunto: Re: [ceph-users] Bucket rename with > > On 2/4/20 12:29 PM, EDH - Manuel Rios wrote: > > Hi > > > > Some Customer asked us for a normal easy problem, they want rename a bucket. > > > > Checking the Nautilus documentation looks by now its not possible, but I checked master documentation and a CLI should be accomplish this apparently. > > > > $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny > > > > Will be backported to Nautilus? Or its still just for developer/master users? > > > > https://docs.ceph.com/docs/master/man/8/radosgw-admin/ > Given both that it's a feature and the sheer size of the PR -- 22 commits and 32 files altered -- my guess is that it will not be backported to Nautilus. However I'll invite the principals to weigh in. > > Best, > > Eric > > -- > J. Eric Ivancich > he/him/his > Red Hat Storage > Ann Arbor, Michigan, USA > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

1 0

Re: Bucket rename with
by J. Eric Ivancich 14 Feb '20

14 Feb '20

On 2/4/20 12:29 PM, EDH - Manuel Rios wrote: > Hi > > Some Customer asked us for a normal easy problem, they want rename a bucket. > > Checking the Nautilus documentation looks by now its not possible, but I checked master documentation and a CLI should be accomplish this apparently. > > $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny > > Will be backported to Nautilus? Or its still just for developer/master users? > > https://docs.ceph.com/docs/master/man/8/radosgw-admin/ Given both that it's a feature and the sheer size of the PR -- 22 commits and 32 files altered -- my guess is that it will not be backported to Nautilus. However I'll invite the principals to weigh in. Best, Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA

2 1

Very bad performance on a ceph rbd pool via iSCSI to VMware esx
by Salsa 14 Feb '20

14 Feb '20

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa

3 4

Monitor / MDS distribution over WAN
by Brian Topping 14 Feb '20

14 Feb '20

I had posted about some of this a year ago in [1] and got some really helpful answers. Fortunately, I know a lot more now and feel a lot more comfortable with the scenario. Because I didn’t understand the architecture very well, I took a pause on distributing monitors and MDS over a WAN. I want to try that now. With a hard limit on the production side of the WAN at two machines and a single monitor/MDS, it’s impossible to upgrade that machine without taking the network down. It only has a few hundred PGs, 8 OSDs and a mostly static CRUSH map. WAN latency is 4ms and there’s a 10Ge link between the production machines, so quorum will be maintained in all cases on the production side except during an upgrade. Most importantly, all the OSDs will remain on the production side of the WAN link. It seems like the worst thing that could happen under normal state is the mon/MDS on the non-prod side of the WAN may be a few clocks behind the quorum on production. In an upgrade state, one of the two production machines is taken down and quorum exists across the WAN. Performance on the cluster might be slower as a result, but everything will remain stable with a stable link. Of course, after the upgrade, quorum is returned to the production side and the normal state returns. This seems like a reasonable working model to me. Do others see holes in my logic? Thanks! Brian [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032271.html

1 0

Learning Ceph - Workshop ideas for entry level
by Ignacio Ocampo 14 Feb '20

14 Feb '20

Hi all, A group of friends and my self are documenting a hands-on workshop about Ceph https://github.com/Nafiux/ceph-workshop, for learning purposes. The idea is to provide visibility step-by-step on common scenarios, from basic usage, to disaster and recovery scenarios. We will hold a workshop next weekend, and some of the ideas for learning are: - Configure and learn how to consume Block Storage Devices - Configure and learn how to consume File Systems Storage - Configure and learn how to consume Object Storage - Simulate a disaster and recovery event by killing a node and setting up a new one - Simulate a node migration Any idea or feedback is welcome! From the way we've decided to install Ceph (ceph-ansible), the way we're configuring the cluster, suggestions on basic day-to-day operations we should learn, etc. Thanks for your support! -- Ignacio Ocampo

2 2

Announcing go-ceph v0.2.0
by John Mulligan 14 Feb '20

14 Feb '20

I'm happy to announce the very first formal release of the go-ceph API bindings. https://github.com/ceph/go-ceph/releases/tag/v0.2.0 They aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild and the ceph-csi project has plans to make use of this library. Specific questions, comments, bugs etc are best directed at our github issues tracker. PS. I was directed here to make this announcement. If this is not the appropriate venue or is considered too spammy please let me know off list. Thanks! --- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0