- ceph-users - lists.ceph.io

Nautilus: Monitors not listening on msgrv1
by Julian Fölsch 24 Jun '20

24 Jun '20

Hi, I am currently facing the problem that our Ceph Cluster running Nautilus is only listening on msgrv2 and we are not sure why. This stops us from using block devices via rbd or mounting ceph via the kernel module. Attached[0] you can find the output of 'cat /etc/ceph/ceph.conf', 'ceph mon dump' and 'ceph config dump'. I already asked on IRC and was told that I probably have more success on the mailing list so hopefully someone here also encountered that issue and can help us out. Kind regards, Julian Fölsch -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Stellvertretender Schatzmeister Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

2 2

NFS Ganesha 2.7 in Xenial not available
by Victoria Martinez de la Cruz 23 Jun '20

23 Jun '20

Hi folks, I'm hitting issues with the nfs-ganesha-stable packages [0], the repo url [1] is broken. Is there a known issue for this? Thanks, Victoria [0] https://shaman.ceph.com/repos/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c… [1] https://chacra.ceph.com/r/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c1bac…

3 2

OSD crash with assertion
by Michael Fladischer 23 Jun '20

23 Jun '20

Hi, a lot of our OSD have crashed a few hours ago because of a failed assertion: /build/ceph-15.2.3/src/osd/ECUtil.h: 34: FAILED ceph_assert(stripe_width % stripe_size == 0) Full output here: https://pastebin.com/D1SXzKsK All OSDs are on bluestore and run 15.2.3. I think I messed up when I tried to change an existing EC profile (using --force) for an active EC pool. I already tried to delete the pool and the EC profile and start the OSDs but they keep crashing with the same assertion. Is there a way to at least find out what the values are for stripe_width and stripe_size? Regards, Michael

3 5

Autoscale recommendtion seems to small + it broke my pool...
by Lindsay Mathieson 23 Jun '20

23 Jun '20

Nautilus 14.2.9, setup using Proxmox. * 5 Hosts * 18 OSDs with a mix of disk sizes (3TB, 1TB, 500GB), all bluestore * Pool size = 3, pg_num = 512 According to: https://docs.ceph.com/docs/nautilus/rados/operations/placement-groups/#pres… With 18 OSD's I should be using pg_num=1024, but I actually have it set to 512. However autoscale is recommending pg_num=128 Additionally, I accidentally set autoscale to on, rather than warn, so it started the process. I rapidly got a "Reduced data availability: 2 pgs inactive" warning and io on the pool stopped. I cleared the warning by restarting the effected OSD's for the pg id, but then more cropped up. I only made it stop and restored access to the pool by turning off autoscale and setting pg_num back to 512. Autoscale warning: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE ceph 3239G 3.0 31205G 0.3114 1.0 512 128 warn osd tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 30.47374 - 30 TiB 7.4 TiB 7.4 TiB 28 MiB 23 GiB 23 TiB 24.40 1.00 - root default -5 5.45798 - 5.5 TiB 1.4 TiB 1.3 TiB 4.4 MiB 3.4 GiB 4.1 TiB 24.78 1.02 - host loc 1 hdd 2.72899 0.95001 2.7 TiB 679 GiB 677 GiB 3.1 MiB 1.9 GiB 2.1 TiB 24.31 1.00 136 up osd.1 8 hdd 2.72899 1.00000 2.7 TiB 706 GiB 704 GiB 1.3 MiB 1.5 GiB 2.0 TiB 25.26 1.04 143 up osd.8 -9 1.36449 - 1.4 TiB 431 GiB 429 GiB 1.4 MiB 2.1 GiB 966 GiB 30.84 1.26 - host lod 5 hdd 0.90970 1.00000 932 GiB 291 GiB 290 GiB 1.1 MiB 1.1 GiB 641 GiB 31.20 1.28 59 up osd.5 12 hdd 0.45479 1.00000 466 GiB 140 GiB 139 GiB 293 KiB 1024 MiB 325 GiB 30.12 1.23 28 up osd.12 -11 10.91595 - 11 TiB 2.4 TiB 2.4 TiB 6.9 MiB 5.8 GiB 8.5 TiB 21.94 0.90 - host vnb 6 hdd 2.72899 1.00000 2.7 TiB 613 GiB 612 GiB 2.5 MiB 1.5 GiB 2.1 TiB 21.95 0.90 124 up osd.6 7 hdd 2.72899 1.00000 2.7 TiB 614 GiB 613 GiB 1.7 MiB 1.4 GiB 2.1 TiB 21.98 0.90 124 up osd.7 9 hdd 2.72899 1.00000 2.7 TiB 617 GiB 615 GiB 1.5 MiB 1.4 GiB 2.1 TiB 22.06 0.90 124 up osd.9 17 hdd 2.72899 1.00000 2.7 TiB 608 GiB 607 GiB 1.3 MiB 1.5 GiB 2.1 TiB 21.76 0.89 124 up osd.17 -3 4.54836 - 4.5 TiB 1.3 TiB 1.3 TiB 11 MiB 7.0 GiB 3.3 TiB 27.76 1.14 - host vnh 0 hdd 0.90970 0.95001 932 GiB 220 GiB 219 GiB 3.0 MiB 1021 MiB 711 GiB 23.64 0.97 44 up osd.0 2 hdd 0.90970 0.95001 932 GiB 252 GiB 251 GiB 536 KiB 1023 MiB 679 GiB 27.06 1.11 51 up osd.2 10 hdd 0.54579 1.00000 559 GiB 158 GiB 157 GiB 1.6 MiB 1022 MiB 401 GiB 28.29 1.16 32 up osd.10 11 hdd 0.54579 1.00000 559 GiB 157 GiB 156 GiB 332 KiB 1024 MiB 402 GiB 28.10 1.15 32 up osd.11 14 hdd 0.54579 1.00000 559 GiB 187 GiB 186 GiB 1.1 MiB 1.0 GiB 372 GiB 33.45 1.37 38 up osd.14 15 hdd 0.54579 1.00000 559 GiB 159 GiB 158 GiB 2.0 MiB 1022 MiB 400 GiB 28.51 1.17 32 up osd.15 16 hdd 0.54579 1.00000 559 GiB 159 GiB 158 GiB 2.8 MiB 1021 MiB 400 GiB 28.46 1.17 32 up osd.16 -7 8.18697 - 8.2 TiB 2.0 TiB 2.0 TiB 4.6 MiB 4.5 GiB 6.2 TiB 24.50 1.00 - host vni 3 hdd 2.72899 1.00000 2.7 TiB 670 GiB 669 GiB 1.2 MiB 1.4 GiB 2.1 TiB 23.99 0.98 134 up osd.3 4 hdd 2.72899 1.00000 2.7 TiB 681 GiB 679 GiB 1.6 MiB 1.6 GiB 2.1 TiB 24.36 1.00 136 up osd.4 13 hdd 2.72899 1.00000 2.7 TiB 703 GiB 701 GiB 1.8 MiB 1.5 GiB 2.0 TiB 25.14 1.03 143 up osd.13 TOTAL 30 TiB 7.4 TiB 7.4 TiB 28 MiB 23 GiB 23 TiB 24.40 Should I be reducing the pg_num? is there a way to do it safely? Thanks. -- Lindsay

2 2

radosgw - how to grant read-only access to another user by default
by Paul Choi 23 Jun '20

23 Jun '20

Hi, I'm new to radosgw (learned more about the MDS than I care to...), and it seems like the buckets and objects created by one user cannot be accessed by another user. Is there a way to make any content created by User A accessible (read-only) by User B? From the documentation it looks like this is handled as an S3 permission but I'm not finding an easy/obvious way to do this. Any help would be appreciated. Thanks in advance!

3 2

Re-run ansible to add monitor and RGWs
by Khodayar Doustar 23 Jun '20

23 Jun '20

Hi, I've installed my ceph cluster with ceph-ansible a few months ago. I've just added one monitor and one rgw at that time. So I have 3 nodes, from which one is monitor and rgw and two others only OSD. Now I want to add the other two nodes as monitor and rgw. Can I just modify the ansible host file and re-run the site.yml? I've done some modification in Storage classes, I've added some OSD and uploaded a lot of data up to now. Is it safe to re-run ansible site.yml playbook? I don't want to end with a fresh new cluster! :D Thanks a lot, Khodayar

3 4

nautilus 14.2.9 cluster no bucket auto sharding
by Marcel Kuiper 22 Jun '20

22 Jun '20

I have 3 ceph clusters on nautilus 14.2.9 (same configuration through puppet). 2 of them are autmatically sharding rados buckets, one of them is not When I do radosgw-admin reshard stale-instances list on the cluster where it does not work I get: reshard stale-instances list Resharding disabled in a multisite env, stale instances unlikely from resharding These instances may not be safe to delete. Use --yes-i-really-mean-it to force displaying these instances. The other 2 clusters don't give this warning. They are all single site. The out put of realm list, zonegroup list and zone list for the cluster that fails auto sharding are as follows realm list { "default_info": "e724bd71-31eb-45c8-a456-151f6a5aa8b5", "realms": [ "backup" ] } zonegroup list { "default_info": "ce4329ae-2bc8-4117-9b82-271022b223fa", "zonegroups": [ "dc3" ] } zone list { "default_info": "7f9bebd6-a9cf-4006-83b1-ff99391aacc0", "zones": [ "dc3-r1" ] } mgr configuration (default value) rgw_dynamic_resharding true How does ceph determine whether it is single site or multi site? How can I force the automated bucket sharding?? Any help would be much appreciated Marcel

1 0

Is there a way froce sync metadata in a multisite cluster
by 黄明友 22 Jun '20

22 Jun '20

Hi,all: the slave zone show metadata is caught up with master ; but use radosgw-admin bucket list|wc diff master and the slave zone , is not equal. how can I force sync it?

3 3

Ceph and linux multi queue block IO layer
by Bobby 22 Jun '20

22 Jun '20

Hi all, One question please. Does Ceph uses the linux multi queue block IO layer ? BR Bobby

1 0

RGW slowdown over time
by Glen Baars 22 Jun '20

22 Jun '20

Hello Ceph users, We are experiencing an issue with ceph 14.2.9 / RGW Beast frontend. We are seeing this across our two separate clusters. Over a few weeks the qlen and qactive are going up and not returning to zero. At some point we start seeing performance degrade and we need to reboot the services. We are viewing the queue numbers in the perfcounters_dump. In objecter_requests we aren't seeing any request ( apart from very briefly ) We can reproduce the issue by use S3 browser and setting the concurrent downloads to 100. After completing download of ~1000 files, the queue length has incremented by 2-5 and never returns back to zero. Subsequent bulk downloads increase the qlen. We have the following tunables set rgw_bucket_index_max_aio 128 rgw_dns_name <fqdn##> rgw_frontends beast ssl_port=443 ssl_certificate=<CERT##> rgw_max_chunk_size 4194304 rgw_num_rados_handles 16 rgw_thread_pool_size 500 Anyone seen this or have any idea how to further debug? Any additional tuning suggested? 350TB S3 data Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

1 0