- ceph-users - lists.ceph.io

Re: MGRs failing once per day and generally slow response times
by Janek Bevendorff 24 Mar '20

24 Mar '20

Indeed. I just had another MGR go bye-bye. I don't think host clock skew is the problem. On 13/03/2020 15:29, Anthony D'Atri wrote: > Chrony does converge faster, but I doubt this will solve your problem if you don’t have quality peers. Or if it’s not really a time problem. > >> On Mar 13, 2020, at 6:44 AM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: >> >> I replaced ntpd with chronyd and will let you know if it changes anything. Thanks. >> >> >>> On 13/03/2020 06:25, Konstantin Shalygin wrote: >>>> On 3/13/20 12:57 AM, Janek Bevendorff wrote: >>>> NTPd is running, all the nodes have the same time to the second. I don't think that is the problem. >>> As always in such cases - try to switch your ntpd to default EL7 daemon - chronyd. >>> >>> >>> >>> k >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Bauhaus-Universität Weimar Bauhausstr. 9a, Room 308 99423 Weimar, Germany Phone: +49 (0)3643 - 58 3577

1 3

Re: Cephfs mount error 1 = Operation not permitted
by Eugen Block 24 Mar '20

24 Mar '20

> I suppose the correct syntax is that anything after "client." is the > name? So: > > ceph fs authorize cephfs client.bob / r / rw > > Would authorize a client named bob? Yes, exactly: admin:~ # ceph fs authorize cephfs client.bob / r / rw [client.bob] key = AQAyw3leAv9tKxAA+wtNEa40yK6svPE/VPlqdA== admin:~ # mount -t ceph mon1:/ /mnt/ -o name=bob,secret=AQAyw3leAv9tKxAA+wtNEa40yK6svPE/VPlqdA== admin:~ # touch /mnt/file Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > That was it! I am not sure how I got confused with the client name > syntax. When I issued the command to create a client key, I used: > > ceph fs authorize cephfs client.1 / r / rw > > I assumed from the syntax that my client name is "client.1" > > I suppose the correct syntax is that anything after "client." is the > name? So: > > ceph fs authorize cephfs client.bob / r / rw > > Would authorize a client named bob? > > -Scott > ________________________________ > From: Eugen Block <eblock(a)nde.ag> > Sent: Monday, March 23, 2020 11:30 AM > To: Dungan, Scott A. <sdungan(a)caltech.edu> > Cc: Yan, Zheng <ukernel(a)gmail.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted > > Wait, your client name is just "1"? In that case you need to specify > that in your mount command: > > mount ... -o name=1,secret=... > > It has to match your ceph auth settings, where "client" is only a > prefix and is followed by the client's name > > [client.1] > > > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > >> Tried that: >> >> [client.1] >> key = ******************************* >> caps mds = "allow rw path=/" >> caps mon = "allow r" >> caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" >> >> No change. >> >> >> ________________________________ >> From: Yan, Zheng <ukernel(a)gmail.com> >> Sent: Sunday, March 22, 2020 9:28 PM >> To: Dungan, Scott A. <sdungan(a)caltech.edu> >> Cc: Eugen Block <eblock(a)nde.ag>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> >> Subject: Re: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted >> >> On Sun, Mar 22, 2020 at 8:21 AM Dungan, Scott A. >> <sdungan(a)caltech.edu> wrote: >>> >>> Zitat, thanks for the tips. >>> >>> I tried appending the key directly in the mount command >>> (secret=<CLIENT.1.SECRET>) and that produced the same error. >>> >>> I took a look at the thread you suggested and I ran the commands >>> that Paul at Croit suggested even though I the ceph dashboard >>> showed "cephs" as already set as the application on both my data >>> and metadata pools: >>> >>> [root@ceph-n4 ~]# ceph osd pool application set data cephfs data cephfs >>> set application 'cephfs' key 'data' to 'cephfs' on pool 'data' >>> [root@ceph-n4 ~]# ceph osd pool application set meta_data cephfs >>> metadata cephfs >>> set application 'cephfs' key 'metadata' to 'cephfs' on pool 'meta_data' >>> >>> No change. I get the "mount error 1 = Operation not permitted" >>> error the same as before. >>> >>> I also tried manually editing the caps osd pool tags for my >>> client.1, to allow rw to both the data pool as well as the metadata >>> pool, as suggested further in the thread: >>> >>> [client.1] >>> key = *********************************** >>> caps mds = "allow rw path=all" >> >> >> try replacing this with "allow rw path=/" >> >>> caps mon = "allow r" >>> caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" >>> >>> No change. >>> >>> ________________________________ >>> From: Eugen Block <eblock(a)nde.ag> >>> Sent: Saturday, March 21, 2020 1:16 PM >>> To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> >>> Subject: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted >>> >>> I just remembered there was a thread [1] about that a couple of weeks >>> ago. Seems like you need to add the capabilities to the client. >>> >>> [1] >>> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… >>> >>> >>> Zitat von Eugen Block <eblock(a)nde.ag>: >>> >>> > Hi, >>> > >>> > have you tried to mount with the secret only instead of a secret file? >>> > >>> > mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=<SECRET> >>> > >>> > If that works your secret file is not right. If not you should check >>> > if the client actually has access to the cephfs pools ('ceph auth >>> > list'). >>> > >>> > >>> > >>> > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: >>> > >>> >> I am still very new to ceph and I have just set up my first small >>> >> test cluster. I have Cephfs enabled (named cephfs) and everything >>> >> is good in the dashboard. I added an authorized user key for cephfs >>> >> with: >>> >> >>> >> ceph fs authorize cephfs client.1 / r / rw >>> >> >>> >> I then copied the key to a file with: >>> >> >>> >> ceph auth get-key client.1 > /tmp/client.1.secret >>> >> >>> >> Copied the file over to the client and then attempt mount witth the >>> >> kernel driver: >>> >> >>> >> mount -t ceph ceph-n4:6789:/ /ceph -o >>> >> name=client.1,secretfile=/root/client.1.secret >>> >> mount error 1 = Operation not permitted >>> >> >>> >> I looked in the logs on the mds (which is also the mgr and mon for >>> >> the cluster) and I don't see any events logged for this. I also >>> >> tried the mount command with verbose and I didn't get any further >>> >> detail. Any tips would be most appreciated. >>> >> >>> >> -- >>> >> >>> >> Scott Dungan >>> >> California Institute of Technology >>> >> Office: (626) 395-3170 >>> >> sdungan(a)caltech.edu<mailto:sdungan@caltech.edu> >>> >> >>> >> _______________________________________________ >>> >> ceph-users mailing list -- ceph-users(a)ceph.io >>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Building a Ceph cluster with Ubuntu 18.04 and NVMe SSDs
by Georg Schönberger 24 Mar '20

24 Mar '20

Hi Ceph users! We are currently configuring our new production Ceph cluster and I have some questions regarding Ubuntu and NVMe SSDs. Basic setup: - Ubuntu 18.04 with HWE Kernel 5.3 - Deployment via ceph-ansible (Ceph stable "Nautilus") - 5x Nodes with AMD EPYC 7402P CPUs - 25Gbit/s NICs and switches for Ceph private and public network - 4x Intel P4510 2TB NVMe SSDs (all flash) per Node My questions: 1. Should we deploy more than one OSD per NVMe SSD? (as P4510's performance can sustain e.g. 2 OSDs) 2. Does anyone know NVMe specific Linux settings we should enable? 3. Can we use io_uring, if yes how can we enable it? Is it enough to set bluestore_iouring=true? What I know so far: Ad 1: My opinion is to use at least 2 OSDs per NVMEe SSD as the Intel P4510 is fast enough to serve the parallel requests. Please be aware to use the latest firmware version VDV10170 -> with version VDV10131 we had massive stalls on Ceph side! Ad 2: I have already enabled NVMe polling queues, Ubuntu has disabled them by default: Added nvme.poll_queues=1 to /etc/default/grub, then checked /sys/block/nvme1n1/queue/io_poll Cf. https://lore.kernel.org/linux-block/20190318222133.GA24176@localhost.locald… Ad 3: This commit states it should be possible to use io_uring: https://github.com/ceph/ceph/pull/27392 This issue also shows how to set bluestore_iouring=true but it's not clear if any more setup is required, like liburing: https://github.com/axboe/liburing A presentation from Christoph Hellwig shows the advantages: https://www.snia.org/sites/default/files/SDC/2019/presentations/NVMe/Hellwi… Any help and inputs would be appreciated, THX - Georg

1 0

Exporting
by Rhian Resnick 24 Mar '20

24 Mar '20

Evening, We are running into issues exporting a disk image from ceph rbd. When we attempt to export an rbd image in a cache tiered erasure-coded pool on Luminus. All the other disks are working fine but this one is acting up. We have a bit of important data on other disks so obviously want to make sure this doesn't happen to those. [root@ceph-p-mon1 home]# rbd export one/one-177-588-0 one-177-588-0 Exporting image: 8% complete...rbd: error reading from source image at offset 5456789504: (5) Input/output error 2020-03-23 20:11:29.210718 7f2f3effd700 -1 librbd::io::ObjectRequest: 0x7f2f2c128f90 handle_read_object: failed to read from object: (5) Input/output error 2020-03-23 20:11:29.565184 7f2f3e7fc700 -1 librbd::io::ObjectRequest: 0x7f2f280c84d0 handle_read_cache: failed to read from cache: (5) Input/output error Exporting image: 8% complete...failed. rbd: export error: (5) Input/output error Any thoughts would be appreciated. Some info: [root@ceph-p-mon1 home]# rbd info one/one-177-588-0 rbd image 'one-177-588-0': size 58.6GiB in 15000 objects order 22 (4MiB objects) block_name_prefix: rbd_data.84a01279e2a9e3 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: create_timestamp: Fri Apr 20 17:06:09 2018 parent: one/one-177@snap overlap: 2.20GiB [root@ceph-p-mon1 home]# ceph status cluster: id: 6a2e8f21-bca2-492b-8869-eecc995216cc health: HEALTH_OK services: mon: 3 daemons, quorum ceph-p-mon2,ceph-p-mon1,ceph-p-mon3 mgr: ceph-p-mon2(active) mds: cephfsec-1/1/1 up {0=ceph-p-mon2=up:active}, 6 up:standby osd: 155 osds: 154 up, 154 in data: pools: 6 pools, 5904 pgs objects: 145.53M objects, 192TiB usage: 253TiB used, 290TiB / 543TiB avail pgs: 5896 active+clean 8 active+clean+scrubbing+deep io: client: 921KiB/s rd, 5.68MiB/s wr, 110op/s rd, 29op/s wr cache: 5.65MiB/s flush, 0op/s promote [root@ceph-p-mon1 home]# rpm -qa | grep ceph ceph-common-12.2.9-0.el7.x86_64 ceph-mds-12.2.9-0.el7.x86_64 ceph-radosgw-12.2.9-0.el7.x86_64 ceph-mgr-12.2.9-0.el7.x86_64 ceph-12.2.9-0.el7.x86_64 collectd-ceph-5.8.1-1.el7.x86_64 ceph-deploy-2.0.1-0.noarch libcephfs2-12.2.9-0.el7.x86_64 python-cephfs-12.2.9-0.el7.x86_64 ceph-selinux-12.2.9-0.el7.x86_64 ceph-osd-12.2.9-0.el7.x86_64 ceph-base-12.2.9-0.el7.x86_64 ceph-mon-12.2.9-0.el7.x86_64 ceph-release-1-1.el7.noarch Rhian Resnick Associate Director Research Computing Enterprise Systems Office of Information Technology Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>

1 0

Docker deploy osd
by Oscar Segarra 24 Mar '20

24 Mar '20

Hi, I'm not able to bootstrap an OSD container for a physical device or LVM. ¿Anyone has been able to bootstrap it? Sorry if it is not the correct place to post this question. If not, I apologize and I will be grateful if anyone can redirect-me to the correct place. Thanks in advance Oscar

1 1

[15.1.1-rc] - "Module 'dashboard' has failed: ('pwdUpdateRequired',)"
by Gencer W. Genç 24 Mar '20

24 Mar '20

After upgrading from 15.1.0 to 15.1.1 of Octopus im seeing this error for dashboard: cluster: id: c5233cbc-e9c2-4db3-85e1-423737a95a8c health: HEALTH_ERR Module 'dashboard' has failed: ('pwdUpdateRequired',) Also executing any command result in: Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: ('pwdUpdateRequired',) How can i fix this problem? By disabling dashboard, Ceph status is back to OK but if i enable ceph dashboard then ERR given. Thanks, Gencer.

4 5

Re: Cephfs mount error 1 = Operation not permitted
by Eugen Block 24 Mar '20

24 Mar '20

Wait, your client name is just "1"? In that case you need to specify that in your mount command: mount ... -o name=1,secret=... It has to match your ceph auth settings, where "client" is only a prefix and is followed by the client's name [client.1] Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > Tried that: > > [client.1] > key = ******************************* > caps mds = "allow rw path=/" > caps mon = "allow r" > caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" > > No change. > > > ________________________________ > From: Yan, Zheng <ukernel(a)gmail.com> > Sent: Sunday, March 22, 2020 9:28 PM > To: Dungan, Scott A. <sdungan(a)caltech.edu> > Cc: Eugen Block <eblock(a)nde.ag>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted > > On Sun, Mar 22, 2020 at 8:21 AM Dungan, Scott A. <sdungan(a)caltech.edu> wrote: >> >> Zitat, thanks for the tips. >> >> I tried appending the key directly in the mount command >> (secret=<CLIENT.1.SECRET>) and that produced the same error. >> >> I took a look at the thread you suggested and I ran the commands >> that Paul at Croit suggested even though I the ceph dashboard >> showed "cephs" as already set as the application on both my data >> and metadata pools: >> >> [root@ceph-n4 ~]# ceph osd pool application set data cephfs data cephfs >> set application 'cephfs' key 'data' to 'cephfs' on pool 'data' >> [root@ceph-n4 ~]# ceph osd pool application set meta_data cephfs >> metadata cephfs >> set application 'cephfs' key 'metadata' to 'cephfs' on pool 'meta_data' >> >> No change. I get the "mount error 1 = Operation not permitted" >> error the same as before. >> >> I also tried manually editing the caps osd pool tags for my >> client.1, to allow rw to both the data pool as well as the metadata >> pool, as suggested further in the thread: >> >> [client.1] >> key = *********************************** >> caps mds = "allow rw path=all" > > > try replacing this with "allow rw path=/" > >> caps mon = "allow r" >> caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" >> >> No change. >> >> ________________________________ >> From: Eugen Block <eblock(a)nde.ag> >> Sent: Saturday, March 21, 2020 1:16 PM >> To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> >> Subject: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted >> >> I just remembered there was a thread [1] about that a couple of weeks >> ago. Seems like you need to add the capabilities to the client. >> >> [1] >> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… >> >> >> Zitat von Eugen Block <eblock(a)nde.ag>: >> >> > Hi, >> > >> > have you tried to mount with the secret only instead of a secret file? >> > >> > mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=<SECRET> >> > >> > If that works your secret file is not right. If not you should check >> > if the client actually has access to the cephfs pools ('ceph auth >> > list'). >> > >> > >> > >> > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: >> > >> >> I am still very new to ceph and I have just set up my first small >> >> test cluster. I have Cephfs enabled (named cephfs) and everything >> >> is good in the dashboard. I added an authorized user key for cephfs >> >> with: >> >> >> >> ceph fs authorize cephfs client.1 / r / rw >> >> >> >> I then copied the key to a file with: >> >> >> >> ceph auth get-key client.1 > /tmp/client.1.secret >> >> >> >> Copied the file over to the client and then attempt mount witth the >> >> kernel driver: >> >> >> >> mount -t ceph ceph-n4:6789:/ /ceph -o >> >> name=client.1,secretfile=/root/client.1.secret >> >> mount error 1 = Operation not permitted >> >> >> >> I looked in the logs on the mds (which is also the mgr and mon for >> >> the cluster) and I don't see any events logged for this. I also >> >> tried the mount command with verbose and I didn't get any further >> >> detail. Any tips would be most appreciated. >> >> >> >> -- >> >> >> >> Scott Dungan >> >> California Institute of Technology >> >> Office: (626) 395-3170 >> >> sdungan(a)caltech.edu<mailto:sdungan@caltech.edu> >> >> >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Cephfs mount error 1 = Operation not permitted
by Yan, Zheng 23 Mar '20

23 Mar '20

On Sun, Mar 22, 2020 at 8:21 AM Dungan, Scott A. <sdungan(a)caltech.edu> wrote: > > Zitat, thanks for the tips. > > I tried appending the key directly in the mount command (secret=<CLIENT.1.SECRET>) and that produced the same error. > > I took a look at the thread you suggested and I ran the commands that Paul at Croit suggested even though I the ceph dashboard showed "cephs" as already set as the application on both my data and metadata pools: > > [root@ceph-n4 ~]# ceph osd pool application set data cephfs data cephfs > set application 'cephfs' key 'data' to 'cephfs' on pool 'data' > [root@ceph-n4 ~]# ceph osd pool application set meta_data cephfs metadata cephfs > set application 'cephfs' key 'metadata' to 'cephfs' on pool 'meta_data' > > No change. I get the "mount error 1 = Operation not permitted" error the same as before. > > I also tried manually editing the caps osd pool tags for my client.1, to allow rw to both the data pool as well as the metadata pool, as suggested further in the thread: > > [client.1] > key = *********************************** > caps mds = "allow rw path=all" try replacing this with "allow rw path=/" > caps mon = "allow r" > caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" > > No change. > > ________________________________ > From: Eugen Block <eblock(a)nde.ag> > Sent: Saturday, March 21, 2020 1:16 PM > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted > > I just remembered there was a thread [1] about that a couple of weeks > ago. Seems like you need to add the capabilities to the client. > > [1] > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… > > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > Hi, > > > > have you tried to mount with the secret only instead of a secret file? > > > > mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=<SECRET> > > > > If that works your secret file is not right. If not you should check > > if the client actually has access to the cephfs pools ('ceph auth > > list'). > > > > > > > > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > > > >> I am still very new to ceph and I have just set up my first small > >> test cluster. I have Cephfs enabled (named cephfs) and everything > >> is good in the dashboard. I added an authorized user key for cephfs > >> with: > >> > >> ceph fs authorize cephfs client.1 / r / rw > >> > >> I then copied the key to a file with: > >> > >> ceph auth get-key client.1 > /tmp/client.1.secret > >> > >> Copied the file over to the client and then attempt mount witth the > >> kernel driver: > >> > >> mount -t ceph ceph-n4:6789:/ /ceph -o > >> name=client.1,secretfile=/root/client.1.secret > >> mount error 1 = Operation not permitted > >> > >> I looked in the logs on the mds (which is also the mgr and mon for > >> the cluster) and I don't see any events logged for this. I also > >> tried the mount command with verbose and I didn't get any further > >> detail. Any tips would be most appreciated. > >> > >> -- > >> > >> Scott Dungan > >> California Institute of Technology > >> Office: (626) 395-3170 > >> sdungan(a)caltech.edu<mailto:sdungan@caltech.edu> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

ceph ignoring cluster/public_network when initiating TCP connections
by Liviu Sas 23 Mar '20

23 Mar '20

Hello, While testing our ceph cluster setup, I noticed a possible issue with the cluster/public network configuration being ignored for TCP session initiation. Looks like the daemons (mon/mgr/mds/osd) are all listening on the right IP address but are initiating TCP sessions from the wrong interfaces. Would it be possible to force ceph daemons to use the cluster/public IP addresses to initiate new TCP connections instead of letting the kernel chose? Some details below: We set everything up to use our "10.2.1.0/24" network: 10.2.1.x (x=node number 1,2,3) But we can see TCP sessions being initiated from "10.2.0.0/24" network. So the daemons are listening to the right IP addresses. root@nbs-vp-01:~# lsof -nPK i | grep ceph | grep LISTE ceph-mds 1541648 ceph 16u IPv4 8169344 0t0 TCP 10.2.1.1:6800 (LISTEN) ceph-mds 1541648 ceph 17u IPv4 8169346 0t0 TCP 10.2.1.1:6801 (LISTEN) ceph-mgr 1541654 ceph 25u IPv4 8163039 0t0 TCP 10.2.1.1:6810 (LISTEN) ceph-mgr 1541654 ceph 27u IPv4 8163051 0t0 TCP 10.2.1.1:6811 (LISTEN) ceph-mon 1541703 ceph 27u IPv4 8170914 0t0 TCP 10.2.1.1:3300 (LISTEN) ceph-mon 1541703 ceph 28u IPv4 8170915 0t0 TCP 10.2.1.1:6789 (LISTEN) ceph-osd 1541711 ceph 16u IPv4 8169353 0t0 TCP 10.2.1.1:6802 (LISTEN) ceph-osd 1541711 ceph 17u IPv4 8169357 0t0 TCP 10.2.1.1:6803 (LISTEN) ceph-osd 1541711 ceph 18u IPv4 8169362 0t0 TCP 10.2.1.1:6804 (LISTEN) ceph-osd 1541711 ceph 19u IPv4 8169368 0t0 TCP 10.2.1.1:6805 (LISTEN) ceph-osd 1541711 ceph 20u IPv4 8169375 0t0 TCP 10.2.1.1:6806 (LISTEN) ceph-osd 1541711 ceph 21u IPv4 8169383 0t0 TCP 10.2.1.1:6807 (LISTEN) ceph-osd 1541711 ceph 22u IPv4 8169392 0t0 TCP 10.2.1.1:6808 (LISTEN) ceph-osd 1541711 ceph 23u IPv4 8169402 0t0 TCP 10.2.1.1:6809 (LISTEN) Sessions to the other nodes use the wrong IP address: @nbs-vp-01:~# lsof -nPK i | grep ceph | grep 10.2.1.2 ceph-mds 1541648 ceph 28u IPv4 8279520 0t0 TCP 10.2.0.2:44180->10.2.1.2:6800 (ESTABLISHED) ceph-mgr 1541654 ceph 41u IPv4 8289842 0t0 TCP 10.2.0.2:44146->10.2.1.2:6800 (ESTABLISHED) ceph-mon 1541703 ceph 40u IPv4 8174827 0t0 TCP 10.2.0.2:40864->10.2.1.2:3300 (ESTABLISHED) ceph-osd 1541711 ceph 65u IPv4 8171035 0t0 TCP 10.2.0.2:58716->10.2.1.2:6804 (ESTABLISHED) ceph-osd 1541711 ceph 66u IPv4 8172960 0t0 TCP 10.2.0.2:54586->10.2.1.2:6806 (ESTABLISHED) root@nbs-vp-01:~# lsof -nPK i | grep ceph | grep 10.2.1.3 ceph-mds 1541648 ceph 30u IPv4 8292421 0t0 TCP 10.2.0.2:45710->10.2.1.3:6802 (ESTABLISHED) ceph-mon 1541703 ceph 46u IPv4 8173025 0t0 TCP 10.2.0.2:40164->10.2.1.3:3300 (ESTABLISHED) ceph-osd 1541711 ceph 67u IPv4 8173043 0t0 TCP 10.2.0.2:56920->10.2.1.3:6804 (ESTABLISHED) ceph-osd 1541711 ceph 68u IPv4 8171063 0t0 TCP 10.2.0.2:41952->10.2.1.3:6806 (ESTABLISHED) ceph-osd 1541711 ceph 69u IPv4 8178891 0t0 TCP 10.2.0.2:57890->10.2.1.3:6808 (ESTABLISHED) See below our cluster config: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.2.1.0/24 fsid = 0f19b6ff-0432-4c3f-b0cb-730e8302dc2c mon_allow_pool_delete = true mon_host = 10.2.1.1 10.2.1.2 10.2.1.3 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.2.1.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.nbs-vp-01] host = nbs-vp-01 mds_standby_for_name = pve [mds.nbs-vp-03] host = nbs-vp-03 mds standby for name = pve [osd.0] public addr = 10.2.1.1 cluster addr = 10.2.1.1 [osd.1] public addr = 10.2.1.2 cluster addr = 10.2.1.2 [osd.2] public addr = 10.2.1.3 cluster addr = 10.2.1.3 [mgr.nbs-vp-01] public addr = 10.2.1.1 [mgr.nbs-vp-02] public addr = 10.2.1.2 [mgr.nbs-vp-03] public addr = 10.2.1.3 [mon.nbs-vp-01] public addr = 10.2.1.1 [mon.nbs-vp-02] public addr = 10.2.1.2 [mon.nbs-vp-03] public addr = 10.2.1.3 Cheers, Liviu

4 5

Q release name
by Sage Weil 23 Mar '20

23 Mar '20

Hi everyone, As we wrap up Octopus and kick of development for Pacific, now it seems like a good idea to sort out what to call the Q release. Traditionally/historically, these have always been names of cephalopod species--usually the "common name", but occasionally a latin name (infernalis). Q is a bit of a challenge since there aren't many of either that start with Q. Nick Barcet found one: quebecoceras, an extinct genus of nautilus (https://en.wikipedia.org/wiki/Quebecoceras) The only other Q cephalopod reference I could find was Squidward Q Tentacles, a character (octopus, strangely) from Spongebob Squarepants, and Yehuda figured out that the Q stands for Quincy. So far that's it. If you can find any other options, please catalog them on the etherpad: https://pad.ceph.com/p/q (or even get a head start on future releases.. they're always the single-letter pads, e.g., https://pad.ceph.com/p/r) sage

9 9