- ceph-users - lists.ceph.io

which is definitive: /var/lib/ceph symlinks or ceph-volume?
by Philip Brown 07 Apr '21

07 Apr '21

I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db On the other hand, ceph-volume lvm list claims that there is a separate db device registered for osd 7 how can I know which one is correct? (This is currently ceph nautilus) -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 5

Increase of osd space usage on cephfs heavy load
by Olivier AUDRY 06 Apr '21

06 Apr '21

hello when I run my borgbackup over cephfs volume (10 subvolumes for 1.5To) I can see a big increase of osd space usage and 2 or 3 osd goes near full, or full, then out and finally the cluster goes in error. Any tips to prevent this ? My cluster is cephv15 with : 9 nodes : each node run : 2x6to hdd and 2x600to ssd the cephfs got data on hdd and metadata on ssd. the cephfs md cache is : 32Go 128pg for data and metadata (this is has been setup by auto balancer) Perhaps I can fix the pg num for each of cephfs pool and prevent autobalancer to run for them. what do you think ? thx you for your help and advices. UPDATE : I increase the pg number to 256 for data and 1024 for metadata Here the df during the backup started since 30min POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs-metadata 12 183 GiB 514.68k 550 GiB 7.16 2.3 TiB Before the backup the stored was 20GiB oau

2 3

Changing IP addresses
by Jean-Marc FONTANA 06 Apr '21

06 Apr '21

Hello everyone, We have installed a Nautilus Ceph cluster with 3 monitors, 5 osd and 1 RGW gateway. It works but now, we need to change the IP addresses of these machines to put them in DMZ. Are there any recommandations to go about doing this ? Best regards,

2 1

Problem using advanced OSD layout in octopus
by Gary Molenkamp 06 Apr '21

06 Apr '21

Before I upgrade our Nautilus ceph to Octopus, I would like to make sure I am able to replace existing OSDs when they fail. However, I have not been able to create an OSD in Octopus with the layout we are using in Nautilus. I am testing this on a VM cluster so as not to touch any production systems. Our existing servers partition one SSD as block_db for 4 HDD OSDs. ie: OST.0 block: /dev/sda, block.db: /dev/sdn1 OST.1 block: /dev/sdb, block.db: /dev/sdn2 OST.2 block: /dev/sdc, block.db: /dev/sdn3 OST.3 block: /dev/sdd, block.db: /dev/sdn4 etc. My understanding is that I will need to apply an advanced osd service to achieve this layout. As each server is is similar the above, I could create 4 services (osd.disk0 - osd.disk3) and apply it to each host. I tried something similar to this: service_type: osd service_id: disk0 placement: host_pattern: 'storage*' data_devices: paths: - /dev/sda db_devices: paths: - /dev/sdn1 But the yaml was rejected with "Exception: Failed to validate Drive Group: `paths` is only allowed for data_devices", although it appears to be valid in the data structures here: https://docs.ceph.com/en/latest/cephadm/osd/#deploy-osds https://people.redhat.com/bhubbard/nature/default/mgr/orchestrator_modules/ I tried to use a combination of size and db_slots for the db_device, but I could not get the OSD to put the block.db on the separate device. Is this possible using the advanced placement method to orchestration, or should I just focus on using "cephadm ceph-volume" to create the OSDs in the desired layout? NOTE: I am trying to avoid installing ceph directly on the host OS (to use ceph-volume) as I do like the containerized approach in octopus. Thank you, Gary -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566

2 2

bug in ceph-volume create
by Philip Brown 06 Apr '21

06 Apr '21

I would file this as a potential bug.. but it takes too long to get approved, and tracker.ceph.com doesnt have straightfoward google signin enabled :-/ I believe that with the new lvm mandate, ceph-volume should not be complaining about "missing PARTUUID". This is stopping me from using my system. Details on how to recreate: 1. have a system with 1 SSD and multiple HDDS 2. create a buncha OSDs with your preferred frontend, which will eventualy come down to ceph-volume lvm batch --bluestore /dev/ssddevice /dev/sdA ... /dev/sdX THIS will work great. batch mode will appropriately carve up the SSD device into multiple LVMs, and allocate one of them to be a DB device for each of the HDDs. 3. try to repair/replace an HDD As soon as you have an HDD fail... you will need to recreate the OSD.. and you are then stuck. Because you cant use batch mode for it... and you cant do it more granularly, with ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdg --block.db /dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here because ceph-volume will complain that, blkid could not detect a PARTUUID for device: /dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here but the lvm IS NOT SUPPOSED TO HAVE A PARTUUID. Which is provable first all by the fact that it isnt a partition. But secondly, that none of the other block-db LVMs it created on the SSD in batch mode, have an partuuid either!! So kindly quit checking for something that isnt supposed to be there in the first place?! (This is a bug all the way back in nautilus, through latest, I believe) -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

3 3

mkfs.xfs -f /dev/rbd0 hangs
by 展荣臻（信泰） 06 Apr '21

06 Apr '21

hi,all: I use 15.2.10 ceph cluster with ubuntu 18.04.I create a rbd device and map into a host（ubuntu 18.04 ceph 15.2.10）, When i type mkfs.fs -f /dev/rbd0 it is hangs,But it is ok that write data use "rados -p {poolname} put obj myfile". Has anyone encountered this kind of problem?

1 0

What is the upper limit of the numer of PGs in a ceph cluster
by huxiaoyu＠horebdata.cn 06 Apr '21

06 Apr '21

Dear Cepher, My client has 10 volume, each volume was assigned 8192 PGs, in total 81920 PGs. The ceph is with Luminous Bluestore. During a power outage, the cluster restarted, and we observed that OSD peering consumed a lot of CPU and memory resources, evne leading to some OSD flappings. My question is thus, 1) how to speed up OSD peering and avoid OSD flappings when there are a lot PGs in the Ceph cluster, and 2) Is there a practical limit on the number of PGs for a single cluster? best regards, Samuel huxiaoyu(a)horebdata.cn

1 0

RGW: Corrupted Bucket index with nautilus 14.2.16
by by morphin 06 Apr '21

06 Apr '21

Good morning. I have a bucket and it has 50M object in it. The bucket created with multisite sync and that is the masterzone and only zone now. After a health check, I saw weird objects in pending attr state. I've tried to remove them with "radosgw-admin object rm --bypas-gc" but I coldn't delete them. After sending delete command I see object attributes changing to pending state but it stays like that. Also I've tried rados rm and the object data is deleted and can't see with radosgw-admin stat. After that I tried to copy the object with rclone "source to dest" But I can not write the object in same path! Rclone gives error but the object is created anyway and I see 0 byte object with rados and radosgw-admin again! After that I've tried to copy the problematic object from /samebucket/objectpath/theObject --> /samebucket/theObject and the 0 byte object copied with success and its not 0 byte anymore! It looks like ok! Then after all these I wonder if its an index issue? I've run radosgw-admin bi list and I saw 29 entry for same object (I guess all these retries left most of them behind) Also I run bi list for a normal object and I see only 5 entry. (Versions). There is bi list for problematic object: https://paste.ubuntu.com/p/jRtH3cMC94/ I'm too scared of checking or re-creating with bucket index. Is there anyone have seen the same issue?

1 0

Pacific unable to configure NFS-Ganesha
by Robert Sander 05 Apr '21

05 Apr '21

Hi, I have a test cluster now running on Pacific with the cephadm orchestrator and upstream container images. In the Dashboard on the services tab I created a new service for NFS. The containers got deployed. But when I go to the NFS tab and try to create a new NFS share the Dashboard only returns a 500 error: Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.146+0000 7f64468d1700 0 [dashboard ERROR exception] Internal Server Error Apr 05 19:38:49 ceph01 bash[35064]: Traceback (most recent call last): Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler Apr 05 19:38:49 ceph01 bash[35064]: return handler(*args, **kwargs) Apr 05 19:38:49 ceph01 bash[35064]: File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ Apr 05 19:38:49 ceph01 bash[35064]: return self.callable(*self.args, **self.kwargs) Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner Apr 05 19:38:49 ceph01 bash[35064]: ret = func(*args, **kwargs) Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/controllers/nfsganesha.py", line 265, in fsals Apr 05 19:38:49 ceph01 bash[35064]: return Ganesha.fsals_available() Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/ganesha.py", line 154, in fsals_available Apr 05 19:38:49 ceph01 bash[35064]: if RgwClient.admin_instance().is_service_online() and \ Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 301, in admin_instance Apr 05 19:38:49 ceph01 bash[35064]: return RgwClient.instance(daemon_name=daemon_name) Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 241, in instance Apr 05 19:38:49 ceph01 bash[35064]: RgwClient._daemons = _get_daemons() Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 53, in _get_daemons Apr 05 19:38:49 ceph01 bash[35064]: raise NoRgwDaemonsException Apr 05 19:38:49 ceph01 bash[35064]: dashboard.services.rgw_client.NoRgwDaemonsException: No RGW service is running. Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+0000 7f64468d1700 0 [dashboard ERROR request] [::ffff:10.0.44.42:39898] [GET] [500] [0.030s] [admin] [513.0B] /ui-api/nfs-ganesha/fsals Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+0000 7f64468d1700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "e89b8519-352f-4e44-a364-6e6faf9dc533"} '] I have no radosgateways in that cluster (currently). There are the pools for radosgw (.rgw.root etc) but no running instance. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 2

Cephfs: Migrating Data to a new Data Pool
by ceph＠fionera.de 05 Apr '21

05 Apr '21

Hello everyone, I currently have a CephFS running with about 60TB of Data. I created it with a replicated pool as default pool, an erasure coded one as additional data pool like it is described in the docs. Now I want to migrate the data from the replicated pool, to the new erasure coded one. I couldn't find any docs and was wondering if its even possible currently. Thank you very much, Fionera

3 4