- ceph-users - lists.ceph.io

Change MON IP in containerized environment
by Eugen Block 18 Dec '20

18 Dec '20

Hi Cephers, there have been several threads in this mailing list about changing IP addresses (especially MON services) and the docs [1] are based on pre-container deployments. I tried to achieve the same for an Octopus cluster (with podman). The "right way" mentioned in the docs is quite easy so I focused on the "messy way". I wrote a blog post [2] about it, maybe it is helpful to some of you. Best regards, Eugen [1] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a… [2] https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors…

1 0

Setting up NFS with Octopus
by Jens Hyllegaard (Soft Design A/S) 18 Dec '20

18 Dec '20

Hi. We are completely new to Ceph, and are exploring using it as an NFS server at first and expand from there. However we have not been successful in getting a working solution. I have set up a test environment with 3 physical servers, each with one OSD using the guide at: https://docs.ceph.com/en/latest/cephadm/install/ I created a new replicated pool: ceph osd pool create objpool replicated And then I deployed the gateway: ceph orch apply nfs objstore objpool nfs-ns I then created a new CephFS volume: ceph fs volume create objstore So far so good 😊 My problem is when I try to create the NFS export The settings are as follows: Cluster: objstore Daemons: nfs.objstore Storage Backend: CephFS CephFS User ID: admin CephFS Name: objstore CephFS Path: /objstore NFS Protocol: NFSV3 Access Type: RW Squash: all_squash Transport protocol: both UDP & TCP Client: Any client can access However when I click on Create NFS export, I get: Failed to create NFS 'objstore:/objstore' error in mkdirs /objstore: Permission denied [Errno 13] Has anyone got an idea as to why this is not working? If you need any further information, do not hesitate to say so. Best regards, Jens Hyllegaard Senior consultant Soft Design Rosenkaeret 13 | DK-2860 Søborg | Denmark | +45 39 66 02 00 | softdesign.dk<http://www.softdesign.dk/> | synchronicer.com

2 1

Erasure Space not showing on Octopus
by Jeremi Avenant 18 Dec '20

18 Dec '20

Good day I currently have a problem where my octopus cluster shows cephfs EC free space differently from my luminous cluster cephfs EC datapool. The only difference I notice is the get application per pool. Mounting the volume from a vm in production Luminous 12.2.13 EC3+1: 10.102.25.18:6789,10.102.25.19:6789,10.102.25.28:6789:/volumes/_nogroup/6f485332-da3d-4f6d-b5aa-a68e9566d1dc 2.8P 2.5P 327T 89% /ilifu 2.8P EC available space 2.5P EC used space 327T available on the EC cephfs_data ceph osd pool application get cephfs_data { "cephfs": {} } Mounting the volume from a vm in production Octopus 15.2.8 EC8+2: 10.102.36.3:6789,10.102.36.5:6789,10.102.36.7:6789:/volumes/_nogroup/<longno> 4.3P 20T 4.3P 1% /new 4.3P is the RAW space of the whole of the ceph cluster, I'm expecting 3.0P Usable Space as show on ceph df cephfs_data 14 16 422 B 3 422 B 0 3.0 PiB ceph osd pool application get cephfs_data { "cephfs": { "data": "cephfs" } } If anyone has an idea how to get the EC volume to show the EC space and not the raw space. Thank you -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

1 0

v14.2.15 Nautilus released
by David Galloway 18 Dec '20

18 Dec '20

This is the 15th backport release in the Nautilus series. This release fixes a ceph-volume regression introduced in v14.2.13 and includes few other fixes. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-15-nautilus-released Notable Changes --------------- * ceph-volume: Fixes lvm batch --auto, which breaks backward compatibility when using non rotational devices only (SSD and/or NVMe). * BlueStore: Fixes a bug in collection_list_legacy which makes pgs inconsistent during scrub when running mixed versions of osds, prior to 14.2.12 with newer. * MGR: progress module can now be turned on/off, using the commands: `ceph progress on` and `ceph progress off`. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.15.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: afdd217ae5fb1ed3f60e16bd62357ca58cc650e5

4 4

Bucket operations an issue with C# AWSSDK.S3 client
by Szabo, Istvan (Agoda) 18 Dec '20

18 Dec '20

Dear Community, We are having issues with bucket operations with ceph octopus 15.2.7. The client library used is : AWSSDK.S3 Version 3.5.6.6 Also used an old version of client: AWSSDK Version 2.3.55.2 Used it in both .net core and simple .net projects but the same result # Note : All methods mentioned below work perfectly fine with nodejs client Node js Client: https://www.npmjs.com/package/ceph Nodejs client doc: https://github.com/YounGoat/nodejs.osapi/blob/2f9d82092589bb50e452c57131499… # ALSO Note : All methods mentioned below work alright with another old version of ceph (luminous 12.2.8) with the same c# client Here are the details: The client object below is new AmazonS3Client( "<Our_Access_Key>", "<Our_Secret_Key>", new AmazonS3Config {ServiceURL = "<Our_Service_Url>" } ); 1. List Buckets. Result: OK Library method used : client.ListBucketsAsync(); [cid:image001.png@01D6D465.D7957510] 1. Create a New Bucket Result: Error 405 MethodNotAllowed (You might think it's a permission issue but it's not, because the same creds work in nodejs) Library method used : var request = new PutBucketRequest {BucketName = "seo"}; // seo is an existing bucket created by command line client.PutBucketAsync(request); Result error snapshot: [cid:image002.png@01D6D465.D7957510] 1. Create Object in an existing bucket Result: Error 501 Not Implemented Library method used : var request = new PutObjectRequest { BucketName = "seo", Key = "test2", ContentType = "text/plain", ContentBody = value, }; client.PutObjectAsync(request); We tried adding headers for content length as well but does not work as well Result error snapshot: [cid:image003.png@01D6D465.D7957510] 1. Read object existing in a bucket Existing bucket name: seo Existing object name(key): test Result: No Such Bucket Library code method: var request = new GetObjectRequest {BucketName = "seo", Key = "test"}; client.GetObjectAsync(request); Result error snapshot: [cid:image004.png@01D6D465.D7957510] Thank you in advance your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 1

changing OSD IP addresses in octopus/docker environment
by Philip Brown 18 Dec '20

18 Dec '20

I was wondering how to change the IPs used for the OSD servers, in my new Octopus based environment, which uses all those docker/podman images by default. imiting date range to within a year, doesnt seem to hit anything. unlimited google search pulled up http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/020503.h… but that references editing /etc/ceph/ceph.conf and changing the [osd.x] sections. which dont exist with Octopus, as far as I've found so far. It doesnt exist in the top level host's /etc/ceph/ceph.conf nor does it exist in the container's conf file, as viewed via "cephadm shell" So, what are the options here? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 1

MDS Corruption: ceph_assert(!p) in MDCache::add_inode
by Brandon Lyon 18 Dec '20

18 Dec '20

This is attempt #3 to submit this issue to this mailing list. I don't expect this to be received. I give up. I have an issue with MDS corruption which so far I haven't been able to resolve using the recovery steps I've found online. I'm on v15.2.6. I've tried all the recovery steps mentioned here, except copying the pool: https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ When I try to start an MDS instance, it crashes after a few seconds. It logs a bunch of "bad backtrace on directory inode" errors before failing on an assertion in MDCache::add_inode, line 313: https://github.com/ceph/ceph/blob/cb8c61a60551b72614257d632a574d420064c17a/… Here's the output of journalctl -xe: https://pastebin.com/9g1UJaKQ I asked in the IRC channel, and it was suggested I might be able to manually delete the duplicate inodes using the RADOS API, though I don't know specifically how I would do that. I have also cloned the code and built Ceph with the problem assertion replaced with a return, but I haven't tried using it yet and I'm saving that as my last resort. I'd appreciate any help you all can give. Thank you, - Brandon Lyon

1 0

bug? cant turn off rbd cache?
by Philip Brown 17 Dec '20

17 Dec '20

Oops.. sent this to the "wrong" list previously. (lists.ceph.com) Lets try the proper one this time :-/ not sure if this an actual bug or I'm doing something else wrong. but in Octopus, I have on the master node # ceph --version ceph version 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) octopus (stable) # tail -3 /etc/ceph/ceph.conf [client] rbd cache = false rbd cache writethrough until flush = false but I have restarted ALL nodes.. and yet rbd cache is still on. # ceph --admin-daemon `find /var/run/ceph -name 'ceph-mon*'` config show |grep rbd_cache "rbd_cache": "true", What else am I supposed to do??? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

3 8

performance degredation every 30 seconds
by Philip Brown 17 Dec '20

17 Dec '20

I have a new 3 node octopus cluster, set up on SSDs. I'm running fio to benchmark the setup, with fio --filename=/dev/rbd0 --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --numjobs=1 --time_based --group_reporting --name=iops-test-job --runtime=120 --eta-newline=1 However, I notice that, approximately every 30 seconds, performance tanks for a bit. Any ideas on why, and better yet, how to get rid of the problem? Sample debug output below. Notice the transitions at [eta 01m:27s] and [eta 00m:49s] It happens again at [00m:09], but figured I didnt need to redundantly post that. Jobs: 1 (f=1): [m(1)][2.5%][r=43.4MiB/s,w=43.3MiB/s][r=11.1k,w=11.1k IOPS][eta 01m:58s] Jobs: 1 (f=1): [m(1)][4.1%][r=47.3MiB/s,w=47.8MiB/s][r=12.1k,w=12.2k IOPS][eta 01m:56s] Jobs: 1 (f=1): [m(1)][5.8%][r=48.6MiB/s,w=49.3MiB/s][r=12.5k,w=12.6k IOPS][eta 01m:54s] Jobs: 1 (f=1): [m(1)][7.4%][r=52.4MiB/s,w=53.1MiB/s][r=13.4k,w=13.6k IOPS][eta 01m:52s] Jobs: 1 (f=1): [m(1)][9.1%][r=54.7MiB/s,w=54.1MiB/s][r=13.0k,w=13.8k IOPS][eta 01m:50s] Jobs: 1 (f=1): [m(1)][10.7%][r=41.5MiB/s,w=42.6MiB/s][r=10.6k,w=10.9k IOPS][eta 01m:48s] Jobs: 1 (f=1): [m(1)][12.4%][r=51.5MiB/s,w=50.6MiB/s][r=13.2k,w=12.0k IOPS][eta 01m:46s] Jobs: 1 (f=1): [m(1)][14.0%][r=16.6MiB/s,w=16.0MiB/s][r=4248,w=4098 IOPS][eta 01m:44s] Jobs: 1 (f=1): [m(1)][14.9%][r=33.3MiB/s,w=33.5MiB/s][r=8526,w=8579 IOPS][eta 01m:43s] Jobs: 1 (f=1): [m(1)][16.5%][r=47.1MiB/s,w=47.4MiB/s][r=12.1k,w=12.1k IOPS][eta 01m:41s] Jobs: 1 (f=1): [m(1)][18.2%][r=49.6MiB/s,w=49.0MiB/s][r=12.7k,w=12.8k IOPS][eta 01m:39s] Jobs: 1 (f=1): [m(1)][19.8%][r=50.3MiB/s,w=51.4MiB/s][r=12.9k,w=13.1k IOPS][eta 01m:37s] Jobs: 1 (f=1): [m(1)][21.5%][r=53.5MiB/s,w=52.9MiB/s][r=13.7k,w=13.5k IOPS][eta 01m:35s] Jobs: 1 (f=1): [m(1)][23.1%][r=52.7MiB/s,w=52.1MiB/s][r=13.5k,w=13.3k IOPS][eta 01m:33s] Jobs: 1 (f=1): [m(1)][24.8%][r=55.3MiB/s,w=54.9MiB/s][r=14.1k,w=14.1k IOPS][eta 01m:31s] Jobs: 1 (f=1): [m(1)][26.4%][r=44.0MiB/s,w=45.2MiB/s][r=11.5k,w=11.6k IOPS][eta 01m:29s] Jobs: 1 (f=1): [m(1)][28.1%][r=12.1MiB/s,w=11.8MiB/s][r=3105,w=3011 IOPS][eta 01m:27s] Jobs: 1 (f=1): [m(1)][29.8%][r=16.6MiB/s,w=17.3MiB/s][r=4238,w=4422 IOPS][eta 01m:25s] Jobs: 1 (f=1): [m(1)][31.4%][r=9820KiB/s,w=9516KiB/s][r=2455,w=2379 IOPS][eta 01m:23s] Jobs: 1 (f=1): [m(1)][33.1%][r=6974KiB/s,w=7099KiB/s][r=1743,w=1774 IOPS][eta 01m:21s] Jobs: 1 (f=1): [m(1)][34.7%][r=49.5MiB/s,w=49.2MiB/s][r=12.7k,w=12.6k IOPS][eta 01m:19s] Jobs: 1 (f=1): [m(1)][36.4%][r=49.3MiB/s,w=49.8MiB/s][r=12.6k,w=12.8k IOPS][eta 01m:17s] Jobs: 1 (f=1): [m(1)][38.0%][r=36.4MiB/s,w=35.9MiB/s][r=9326,w=9200 IOPS][eta 01m:15s] Jobs: 1 (f=1): [m(1)][39.7%][r=43.4MiB/s,w=43.3MiB/s][r=11.1k,w=11.1k IOPS][eta 01m:13s] Jobs: 1 (f=1): [m(1)][41.3%][r=47.1MiB/s,w=47.1MiB/s][r=12.1k,w=12.1k IOPS][eta 01m:11s] Jobs: 1 (f=1): [m(1)][43.0%][r=47.9MiB/s,w=48.0MiB/s][r=12.3k,w=12.5k IOPS][eta 01m:09s] Jobs: 1 (f=1): [m(1)][44.6%][r=49.9MiB/s,w=48.8MiB/s][r=12.8k,w=12.5k IOPS][eta 01m:07s] Jobs: 1 (f=1): [m(1)][46.3%][r=46.4MiB/s,w=46.9MiB/s][r=11.9k,w=11.0k IOPS][eta 01m:05s] Jobs: 1 (f=1): [m(1)][47.9%][r=46.7MiB/s,w=46.4MiB/s][r=11.0k,w=11.9k IOPS][eta 01m:03s] Jobs: 1 (f=1): [m(1)][49.6%][r=55.3MiB/s,w=55.3MiB/s][r=14.1k,w=14.2k IOPS][eta 01m:01s] Jobs: 1 (f=1): [m(1)][51.2%][r=54.1MiB/s,w=53.2MiB/s][r=13.8k,w=13.6k IOPS][eta 00m:59s] Jobs: 1 (f=1): [m(1)][52.9%][r=53.4MiB/s,w=52.9MiB/s][r=13.7k,w=13.6k IOPS][eta 00m:57s] Jobs: 1 (f=1): [m(1)][54.5%][r=58.8MiB/s,w=58.0MiB/s][r=15.1k,w=15.1k IOPS][eta 00m:55s] Jobs: 1 (f=1): [m(1)][56.2%][r=60.0MiB/s,w=58.6MiB/s][r=15.4k,w=15.0k IOPS][eta 00m:53s] Jobs: 1 (f=1): [m(1)][57.9%][r=57.7MiB/s,w=58.1MiB/s][r=14.8k,w=14.9k IOPS][eta 00m:51s] Jobs: 1 (f=1): [m(1)][59.5%][r=14.0MiB/s,w=14.3MiB/s][r=3592,w=3651 IOPS][eta 00m:49s] Jobs: 1 (f=1): [m(1)][61.2%][r=17.4MiB/s,w=17.4MiB/s][r=4443,w=4457 IOPS][eta 00m:47s] Jobs: 1 (f=1): [m(1)][62.8%][r=18.1MiB/s,w=18.7MiB/s][r=4640,w=4783 IOPS][eta 00m:45s] Jobs: 1 (f=1): [m(1)][64.5%][r=7896KiB/s,w=8300KiB/s][r=1974,w=2075 IOPS][eta 00m:43s] Jobs: 1 (f=1): [m(1)][66.1%][r=47.8MiB/s,w=47.3MiB/s][r=12.2k,w=12.1k IOPS][eta 00m:41s] -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

5 16

reliability of rados_stat() function
by Peter Lieven 17 Dec '20

17 Dec '20

Hi all, the rados_stat() function has a TODO in the comments: * TODO: when are these set, and by whom? can they be out of date? Can anyone help with this? How reliably is the pmtime updated? Is there a minimum update interval? Thank you, Peter

1 1