June 2020 - ceph-users - lists.ceph.io

by Simon Sutter

Hello, After two months of the "ceph try and error game", I finally managed to get an Octopuss cluster up and running. The unconventional thing about it is, it's just for hot backups, no virtual machines on there. All the nodes are without any caching ssd's, just plain hdd's. At the moment there are eight of them with a total of 50TB. We are planning to go up to 25 and bigger disks so we end on 300TB-400TB I decided to go with cephfs, because I don't have any experience in things like S3 and I need to read the same file system from more than one client. I made one cephfs with a replicated pool. On there I added erasure-coded pools to save some Storage. To add those pools, I did it with the setfattr command like this: setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1 Some of our servers cannot use cephfs (old kernels, special OS's) so I have to use nfs. This is set up with the included ganesha-nfs. Exported is the /cephfs/nfs folder and clients can mount folders below this. There are two final questions: - Was it right to go with the way of "mounting" pools with setfattr, or should I have used multiple cephfs? First I was thinking about using multiple cephfs but there are warnings everywhere. The deeper I got in, the more it seems I would have been fine with multiple cephfs. - Is there a way I don't know, but it would be easier? I still don't know much about Rest, S3, RBD etc... so there may be a better way Other remarks are desired. Thanks in advance, Simon

3 years, 10 months

2
1
0 0

How to ceph-volume on remote hosts?

by steven prothero

Hello, I am new to CEPH and on a few test servers attempting to setup and learn a test ceph system. I started off the install with the "Cephadm" option and it uses podman containers. Followed steps here: https://docs.ceph.com/docs/master/cephadm/install/ I ran the bootstrap, added remote hosts, added monitors and everything is looking good. Now I would like to add OSDs... On the bootstrapped server i did a : ceph-volume lvm prepare --data /dev/sda6 and then the "activate" and "ceph orch daemon add osd (etc)" to add it and it works... But now I am ready to add OSDs on the remote nodes. I am not able to find documentation or examples on how to do : ceph-volume lvm prepare & activate steps on the remote hosts. How do we prepare & activate the remote hosts disks? Thank you very much for your input, Cheers Steve

3 years, 10 months

3
2
0 0

How to remove one of two filesystems

by Francois Legrand

Hello, I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds. mds1 is managing fs1 mds2 manages fs2 mds3 is standby I want to completely remove fs1. It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it and then delete the data and metadata pools with ceph osd pool delete but in many threads I noticed that you must shutdown the mds before running ceph fs rm. Is it still the case ? What happens in my configuration (I have 2 fs) ? If I stop mds1, the mds3 will take the management. If I stop mds3 what will mds2 do (try to manage the 2 fs or continue only with fs2) ? Thanks for your advices. F.

3 years, 10 months

4
7
0 0

Nautilus: Monitors not listening on msgrv1

by Julian Fölsch

Hi, I am currently facing the problem that our Ceph Cluster running Nautilus is only listening on msgrv2 and we are not sure why. This stops us from using block devices via rbd or mounting ceph via the kernel module. Attached[0] you can find the output of 'cat /etc/ceph/ceph.conf', 'ceph mon dump' and 'ceph config dump'. I already asked on IRC and was told that I probably have more success on the mailing list so hopefully someone here also encountered that issue and can help us out. Kind regards, Julian Fölsch -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Stellvertretender Schatzmeister Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

3 years, 10 months

2
2
0 0

NFS Ganesha 2.7 in Xenial not available

by Victoria Martinez de la Cruz

Hi folks, I'm hitting issues with the nfs-ganesha-stable packages [0], the repo url [1] is broken. Is there a known issue for this? Thanks, Victoria [0] https://shaman.ceph.com/repos/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c… [1] https://chacra.ceph.com/r/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c1bac…

3 years, 10 months

3
2
0 0

OSD crash with assertion

by Michael Fladischer

Hi, a lot of our OSD have crashed a few hours ago because of a failed assertion: /build/ceph-15.2.3/src/osd/ECUtil.h: 34: FAILED ceph_assert(stripe_width % stripe_size == 0) Full output here: https://pastebin.com/D1SXzKsK All OSDs are on bluestore and run 15.2.3. I think I messed up when I tried to change an existing EC profile (using --force) for an active EC pool. I already tried to delete the pool and the EC profile and start the OSDs but they keep crashing with the same assertion. Is there a way to at least find out what the values are for stripe_width and stripe_size? Regards, Michael

3 years, 10 months

3
5
0 0

Autoscale recommendtion seems to small + it broke my pool...

by Lindsay Mathieson

Nautilus 14.2.9, setup using Proxmox. * 5 Hosts * 18 OSDs with a mix of disk sizes (3TB, 1TB, 500GB), all bluestore * Pool size = 3, pg_num = 512 According to: https://docs.ceph.com/docs/nautilus/rados/operations/placement-groups/#pres… With 18 OSD's I should be using pg_num=1024, but I actually have it set to 512. However autoscale is recommending pg_num=128 Additionally, I accidentally set autoscale to on, rather than warn, so it started the process. I rapidly got a "Reduced data availability: 2 pgs inactive" warning and io on the pool stopped. I cleared the warning by restarting the effected OSD's for the pg id, but then more cropped up. I only made it stop and restored access to the pool by turning off autoscale and setting pg_num back to 512. Autoscale warning: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE ceph 3239G 3.0 31205G 0.3114 1.0 512 128 warn osd tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 30.47374 - 30 TiB 7.4 TiB 7.4 TiB 28 MiB 23 GiB 23 TiB 24.40 1.00 - root default -5 5.45798 - 5.5 TiB 1.4 TiB 1.3 TiB 4.4 MiB 3.4 GiB 4.1 TiB 24.78 1.02 - host loc 1 hdd 2.72899 0.95001 2.7 TiB 679 GiB 677 GiB 3.1 MiB 1.9 GiB 2.1 TiB 24.31 1.00 136 up osd.1 8 hdd 2.72899 1.00000 2.7 TiB 706 GiB 704 GiB 1.3 MiB 1.5 GiB 2.0 TiB 25.26 1.04 143 up osd.8 -9 1.36449 - 1.4 TiB 431 GiB 429 GiB 1.4 MiB 2.1 GiB 966 GiB 30.84 1.26 - host lod 5 hdd 0.90970 1.00000 932 GiB 291 GiB 290 GiB 1.1 MiB 1.1 GiB 641 GiB 31.20 1.28 59 up osd.5 12 hdd 0.45479 1.00000 466 GiB 140 GiB 139 GiB 293 KiB 1024 MiB 325 GiB 30.12 1.23 28 up osd.12 -11 10.91595 - 11 TiB 2.4 TiB 2.4 TiB 6.9 MiB 5.8 GiB 8.5 TiB 21.94 0.90 - host vnb 6 hdd 2.72899 1.00000 2.7 TiB 613 GiB 612 GiB 2.5 MiB 1.5 GiB 2.1 TiB 21.95 0.90 124 up osd.6 7 hdd 2.72899 1.00000 2.7 TiB 614 GiB 613 GiB 1.7 MiB 1.4 GiB 2.1 TiB 21.98 0.90 124 up osd.7 9 hdd 2.72899 1.00000 2.7 TiB 617 GiB 615 GiB 1.5 MiB 1.4 GiB 2.1 TiB 22.06 0.90 124 up osd.9 17 hdd 2.72899 1.00000 2.7 TiB 608 GiB 607 GiB 1.3 MiB 1.5 GiB 2.1 TiB 21.76 0.89 124 up osd.17 -3 4.54836 - 4.5 TiB 1.3 TiB 1.3 TiB 11 MiB 7.0 GiB 3.3 TiB 27.76 1.14 - host vnh 0 hdd 0.90970 0.95001 932 GiB 220 GiB 219 GiB 3.0 MiB 1021 MiB 711 GiB 23.64 0.97 44 up osd.0 2 hdd 0.90970 0.95001 932 GiB 252 GiB 251 GiB 536 KiB 1023 MiB 679 GiB 27.06 1.11 51 up osd.2 10 hdd 0.54579 1.00000 559 GiB 158 GiB 157 GiB 1.6 MiB 1022 MiB 401 GiB 28.29 1.16 32 up osd.10 11 hdd 0.54579 1.00000 559 GiB 157 GiB 156 GiB 332 KiB 1024 MiB 402 GiB 28.10 1.15 32 up osd.11 14 hdd 0.54579 1.00000 559 GiB 187 GiB 186 GiB 1.1 MiB 1.0 GiB 372 GiB 33.45 1.37 38 up osd.14 15 hdd 0.54579 1.00000 559 GiB 159 GiB 158 GiB 2.0 MiB 1022 MiB 400 GiB 28.51 1.17 32 up osd.15 16 hdd 0.54579 1.00000 559 GiB 159 GiB 158 GiB 2.8 MiB 1021 MiB 400 GiB 28.46 1.17 32 up osd.16 -7 8.18697 - 8.2 TiB 2.0 TiB 2.0 TiB 4.6 MiB 4.5 GiB 6.2 TiB 24.50 1.00 - host vni 3 hdd 2.72899 1.00000 2.7 TiB 670 GiB 669 GiB 1.2 MiB 1.4 GiB 2.1 TiB 23.99 0.98 134 up osd.3 4 hdd 2.72899 1.00000 2.7 TiB 681 GiB 679 GiB 1.6 MiB 1.6 GiB 2.1 TiB 24.36 1.00 136 up osd.4 13 hdd 2.72899 1.00000 2.7 TiB 703 GiB 701 GiB 1.8 MiB 1.5 GiB 2.0 TiB 25.14 1.03 143 up osd.13 TOTAL 30 TiB 7.4 TiB 7.4 TiB 28 MiB 23 GiB 23 TiB 24.40 Should I be reducing the pg_num? is there a way to do it safely? Thanks. -- Lindsay

3 years, 10 months

2
2
0 0

radosgw - how to grant read-only access to another user by default

by Paul Choi

Hi, I'm new to radosgw (learned more about the MDS than I care to...), and it seems like the buckets and objects created by one user cannot be accessed by another user. Is there a way to make any content created by User A accessible (read-only) by User B? From the documentation it looks like this is handled as an S3 permission but I'm not finding an easy/obvious way to do this. Any help would be appreciated. Thanks in advance!

3 years, 10 months

3
2
0 0

Re-run ansible to add monitor and RGWs

by Khodayar Doustar

Hi, I've installed my ceph cluster with ceph-ansible a few months ago. I've just added one monitor and one rgw at that time. So I have 3 nodes, from which one is monitor and rgw and two others only OSD. Now I want to add the other two nodes as monitor and rgw. Can I just modify the ansible host file and re-run the site.yml? I've done some modification in Storage classes, I've added some OSD and uploaded a lot of data up to now. Is it safe to re-run ansible site.yml playbook? I don't want to end with a fresh new cluster! :D Thanks a lot, Khodayar

3 years, 10 months

3
4
0 0

nautilus 14.2.9 cluster no bucket auto sharding

by Marcel Kuiper

I have 3 ceph clusters on nautilus 14.2.9 (same configuration through puppet). 2 of them are autmatically sharding rados buckets, one of them is not When I do radosgw-admin reshard stale-instances list on the cluster where it does not work I get: reshard stale-instances list Resharding disabled in a multisite env, stale instances unlikely from resharding These instances may not be safe to delete. Use --yes-i-really-mean-it to force displaying these instances. The other 2 clusters don't give this warning. They are all single site. The out put of realm list, zonegroup list and zone list for the cluster that fails auto sharding are as follows realm list { "default_info": "e724bd71-31eb-45c8-a456-151f6a5aa8b5", "realms": [ "backup" ] } zonegroup list { "default_info": "ce4329ae-2bc8-4117-9b82-271022b223fa", "zonegroups": [ "dc3" ] } zone list { "default_info": "7f9bebd6-a9cf-4006-83b1-ff99391aacc0", "zones": [ "dc3-r1" ] } mgr configuration (default value) rgw_dynamic_resharding true How does ceph determine whether it is single site or multi site? How can I force the automated bucket sharding?? Any help would be much appreciated Marcel

3 years, 10 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users June 2020