March 2020 - ceph-users - lists.ceph.io

Docker deploy osd
by Oscar Segarra 24 Mar '20

24 Mar '20

Hi, I'm not able to bootstrap an OSD container for a physical device or LVM. ¿Anyone has been able to bootstrap it? Sorry if it is not the correct place to post this question. If not, I apologize and I will be grateful if anyone can redirect-me to the correct place. Thanks in advance Oscar

1 1

[15.1.1-rc] - "Module 'dashboard' has failed: ('pwdUpdateRequired',)"
by Gencer W. Genç 23 Mar '20

23 Mar '20

After upgrading from 15.1.0 to 15.1.1 of Octopus im seeing this error for dashboard: cluster: id: c5233cbc-e9c2-4db3-85e1-423737a95a8c health: HEALTH_ERR Module 'dashboard' has failed: ('pwdUpdateRequired',) Also executing any command result in: Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: ('pwdUpdateRequired',) How can i fix this problem? By disabling dashboard, Ceph status is back to OK but if i enable ceph dashboard then ERR given. Thanks, Gencer.

4 5

Re: Cephfs mount error 1 = Operation not permitted
by Eugen Block 23 Mar '20

23 Mar '20

Wait, your client name is just "1"? In that case you need to specify that in your mount command: mount ... -o name=1,secret=... It has to match your ceph auth settings, where "client" is only a prefix and is followed by the client's name [client.1] Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > Tried that: > > [client.1] > key = ******************************* > caps mds = "allow rw path=/" > caps mon = "allow r" > caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" > > No change. > > > ________________________________ > From: Yan, Zheng <ukernel(a)gmail.com> > Sent: Sunday, March 22, 2020 9:28 PM > To: Dungan, Scott A. <sdungan(a)caltech.edu> > Cc: Eugen Block <eblock(a)nde.ag>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted > > On Sun, Mar 22, 2020 at 8:21 AM Dungan, Scott A. <sdungan(a)caltech.edu> wrote: >> >> Zitat, thanks for the tips. >> >> I tried appending the key directly in the mount command >> (secret=<CLIENT.1.SECRET>) and that produced the same error. >> >> I took a look at the thread you suggested and I ran the commands >> that Paul at Croit suggested even though I the ceph dashboard >> showed "cephs" as already set as the application on both my data >> and metadata pools: >> >> [root@ceph-n4 ~]# ceph osd pool application set data cephfs data cephfs >> set application 'cephfs' key 'data' to 'cephfs' on pool 'data' >> [root@ceph-n4 ~]# ceph osd pool application set meta_data cephfs >> metadata cephfs >> set application 'cephfs' key 'metadata' to 'cephfs' on pool 'meta_data' >> >> No change. I get the "mount error 1 = Operation not permitted" >> error the same as before. >> >> I also tried manually editing the caps osd pool tags for my >> client.1, to allow rw to both the data pool as well as the metadata >> pool, as suggested further in the thread: >> >> [client.1] >> key = *********************************** >> caps mds = "allow rw path=all" > > > try replacing this with "allow rw path=/" > >> caps mon = "allow r" >> caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" >> >> No change. >> >> ________________________________ >> From: Eugen Block <eblock(a)nde.ag> >> Sent: Saturday, March 21, 2020 1:16 PM >> To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> >> Subject: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted >> >> I just remembered there was a thread [1] about that a couple of weeks >> ago. Seems like you need to add the capabilities to the client. >> >> [1] >> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… >> >> >> Zitat von Eugen Block <eblock(a)nde.ag>: >> >> > Hi, >> > >> > have you tried to mount with the secret only instead of a secret file? >> > >> > mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=<SECRET> >> > >> > If that works your secret file is not right. If not you should check >> > if the client actually has access to the cephfs pools ('ceph auth >> > list'). >> > >> > >> > >> > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: >> > >> >> I am still very new to ceph and I have just set up my first small >> >> test cluster. I have Cephfs enabled (named cephfs) and everything >> >> is good in the dashboard. I added an authorized user key for cephfs >> >> with: >> >> >> >> ceph fs authorize cephfs client.1 / r / rw >> >> >> >> I then copied the key to a file with: >> >> >> >> ceph auth get-key client.1 > /tmp/client.1.secret >> >> >> >> Copied the file over to the client and then attempt mount witth the >> >> kernel driver: >> >> >> >> mount -t ceph ceph-n4:6789:/ /ceph -o >> >> name=client.1,secretfile=/root/client.1.secret >> >> mount error 1 = Operation not permitted >> >> >> >> I looked in the logs on the mds (which is also the mgr and mon for >> >> the cluster) and I don't see any events logged for this. I also >> >> tried the mount command with verbose and I didn't get any further >> >> detail. Any tips would be most appreciated. >> >> >> >> -- >> >> >> >> Scott Dungan >> >> California Institute of Technology >> >> Office: (626) 395-3170 >> >> sdungan(a)caltech.edu<mailto:sdungan@caltech.edu> >> >> >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Cephfs mount error 1 = Operation not permitted
by Yan, Zheng 23 Mar '20

23 Mar '20

On Sun, Mar 22, 2020 at 8:21 AM Dungan, Scott A. <sdungan(a)caltech.edu> wrote: > > Zitat, thanks for the tips. > > I tried appending the key directly in the mount command (secret=<CLIENT.1.SECRET>) and that produced the same error. > > I took a look at the thread you suggested and I ran the commands that Paul at Croit suggested even though I the ceph dashboard showed "cephs" as already set as the application on both my data and metadata pools: > > [root@ceph-n4 ~]# ceph osd pool application set data cephfs data cephfs > set application 'cephfs' key 'data' to 'cephfs' on pool 'data' > [root@ceph-n4 ~]# ceph osd pool application set meta_data cephfs metadata cephfs > set application 'cephfs' key 'metadata' to 'cephfs' on pool 'meta_data' > > No change. I get the "mount error 1 = Operation not permitted" error the same as before. > > I also tried manually editing the caps osd pool tags for my client.1, to allow rw to both the data pool as well as the metadata pool, as suggested further in the thread: > > [client.1] > key = *********************************** > caps mds = "allow rw path=all" try replacing this with "allow rw path=/" > caps mon = "allow r" > caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" > > No change. > > ________________________________ > From: Eugen Block <eblock(a)nde.ag> > Sent: Saturday, March 21, 2020 1:16 PM > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted > > I just remembered there was a thread [1] about that a couple of weeks > ago. Seems like you need to add the capabilities to the client. > > [1] > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… > > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > Hi, > > > > have you tried to mount with the secret only instead of a secret file? > > > > mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=<SECRET> > > > > If that works your secret file is not right. If not you should check > > if the client actually has access to the cephfs pools ('ceph auth > > list'). > > > > > > > > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > > > >> I am still very new to ceph and I have just set up my first small > >> test cluster. I have Cephfs enabled (named cephfs) and everything > >> is good in the dashboard. I added an authorized user key for cephfs > >> with: > >> > >> ceph fs authorize cephfs client.1 / r / rw > >> > >> I then copied the key to a file with: > >> > >> ceph auth get-key client.1 > /tmp/client.1.secret > >> > >> Copied the file over to the client and then attempt mount witth the > >> kernel driver: > >> > >> mount -t ceph ceph-n4:6789:/ /ceph -o > >> name=client.1,secretfile=/root/client.1.secret > >> mount error 1 = Operation not permitted > >> > >> I looked in the logs on the mds (which is also the mgr and mon for > >> the cluster) and I don't see any events logged for this. I also > >> tried the mount command with verbose and I didn't get any further > >> detail. Any tips would be most appreciated. > >> > >> -- > >> > >> Scott Dungan > >> California Institute of Technology > >> Office: (626) 395-3170 > >> sdungan(a)caltech.edu<mailto:sdungan@caltech.edu> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

ceph ignoring cluster/public_network when initiating TCP connections
by Liviu Sas 23 Mar '20

23 Mar '20

Hello, While testing our ceph cluster setup, I noticed a possible issue with the cluster/public network configuration being ignored for TCP session initiation. Looks like the daemons (mon/mgr/mds/osd) are all listening on the right IP address but are initiating TCP sessions from the wrong interfaces. Would it be possible to force ceph daemons to use the cluster/public IP addresses to initiate new TCP connections instead of letting the kernel chose? Some details below: We set everything up to use our "10.2.1.0/24" network: 10.2.1.x (x=node number 1,2,3) But we can see TCP sessions being initiated from "10.2.0.0/24" network. So the daemons are listening to the right IP addresses. root@nbs-vp-01:~# lsof -nPK i | grep ceph | grep LISTE ceph-mds 1541648 ceph 16u IPv4 8169344 0t0 TCP 10.2.1.1:6800 (LISTEN) ceph-mds 1541648 ceph 17u IPv4 8169346 0t0 TCP 10.2.1.1:6801 (LISTEN) ceph-mgr 1541654 ceph 25u IPv4 8163039 0t0 TCP 10.2.1.1:6810 (LISTEN) ceph-mgr 1541654 ceph 27u IPv4 8163051 0t0 TCP 10.2.1.1:6811 (LISTEN) ceph-mon 1541703 ceph 27u IPv4 8170914 0t0 TCP 10.2.1.1:3300 (LISTEN) ceph-mon 1541703 ceph 28u IPv4 8170915 0t0 TCP 10.2.1.1:6789 (LISTEN) ceph-osd 1541711 ceph 16u IPv4 8169353 0t0 TCP 10.2.1.1:6802 (LISTEN) ceph-osd 1541711 ceph 17u IPv4 8169357 0t0 TCP 10.2.1.1:6803 (LISTEN) ceph-osd 1541711 ceph 18u IPv4 8169362 0t0 TCP 10.2.1.1:6804 (LISTEN) ceph-osd 1541711 ceph 19u IPv4 8169368 0t0 TCP 10.2.1.1:6805 (LISTEN) ceph-osd 1541711 ceph 20u IPv4 8169375 0t0 TCP 10.2.1.1:6806 (LISTEN) ceph-osd 1541711 ceph 21u IPv4 8169383 0t0 TCP 10.2.1.1:6807 (LISTEN) ceph-osd 1541711 ceph 22u IPv4 8169392 0t0 TCP 10.2.1.1:6808 (LISTEN) ceph-osd 1541711 ceph 23u IPv4 8169402 0t0 TCP 10.2.1.1:6809 (LISTEN) Sessions to the other nodes use the wrong IP address: @nbs-vp-01:~# lsof -nPK i | grep ceph | grep 10.2.1.2 ceph-mds 1541648 ceph 28u IPv4 8279520 0t0 TCP 10.2.0.2:44180->10.2.1.2:6800 (ESTABLISHED) ceph-mgr 1541654 ceph 41u IPv4 8289842 0t0 TCP 10.2.0.2:44146->10.2.1.2:6800 (ESTABLISHED) ceph-mon 1541703 ceph 40u IPv4 8174827 0t0 TCP 10.2.0.2:40864->10.2.1.2:3300 (ESTABLISHED) ceph-osd 1541711 ceph 65u IPv4 8171035 0t0 TCP 10.2.0.2:58716->10.2.1.2:6804 (ESTABLISHED) ceph-osd 1541711 ceph 66u IPv4 8172960 0t0 TCP 10.2.0.2:54586->10.2.1.2:6806 (ESTABLISHED) root@nbs-vp-01:~# lsof -nPK i | grep ceph | grep 10.2.1.3 ceph-mds 1541648 ceph 30u IPv4 8292421 0t0 TCP 10.2.0.2:45710->10.2.1.3:6802 (ESTABLISHED) ceph-mon 1541703 ceph 46u IPv4 8173025 0t0 TCP 10.2.0.2:40164->10.2.1.3:3300 (ESTABLISHED) ceph-osd 1541711 ceph 67u IPv4 8173043 0t0 TCP 10.2.0.2:56920->10.2.1.3:6804 (ESTABLISHED) ceph-osd 1541711 ceph 68u IPv4 8171063 0t0 TCP 10.2.0.2:41952->10.2.1.3:6806 (ESTABLISHED) ceph-osd 1541711 ceph 69u IPv4 8178891 0t0 TCP 10.2.0.2:57890->10.2.1.3:6808 (ESTABLISHED) See below our cluster config: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.2.1.0/24 fsid = 0f19b6ff-0432-4c3f-b0cb-730e8302dc2c mon_allow_pool_delete = true mon_host = 10.2.1.1 10.2.1.2 10.2.1.3 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.2.1.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.nbs-vp-01] host = nbs-vp-01 mds_standby_for_name = pve [mds.nbs-vp-03] host = nbs-vp-03 mds standby for name = pve [osd.0] public addr = 10.2.1.1 cluster addr = 10.2.1.1 [osd.1] public addr = 10.2.1.2 cluster addr = 10.2.1.2 [osd.2] public addr = 10.2.1.3 cluster addr = 10.2.1.3 [mgr.nbs-vp-01] public addr = 10.2.1.1 [mgr.nbs-vp-02] public addr = 10.2.1.2 [mgr.nbs-vp-03] public addr = 10.2.1.3 [mon.nbs-vp-01] public addr = 10.2.1.1 [mon.nbs-vp-02] public addr = 10.2.1.2 [mon.nbs-vp-03] public addr = 10.2.1.3 Cheers, Liviu

4 5

Q release name
by Sage Weil 23 Mar '20

23 Mar '20

Hi everyone, As we wrap up Octopus and kick of development for Pacific, now it seems like a good idea to sort out what to call the Q release. Traditionally/historically, these have always been names of cephalopod species--usually the "common name", but occasionally a latin name (infernalis). Q is a bit of a challenge since there aren't many of either that start with Q. Nick Barcet found one: quebecoceras, an extinct genus of nautilus (https://en.wikipedia.org/wiki/Quebecoceras) The only other Q cephalopod reference I could find was Squidward Q Tentacles, a character (octopus, strangely) from Spongebob Squarepants, and Yehuda figured out that the Q stands for Quincy. So far that's it. If you can find any other options, please catalog them on the etherpad: https://pad.ceph.com/p/q (or even get a head start on future releases.. they're always the single-letter pads, e.g., https://pad.ceph.com/p/r) sage

9 9

OSD: FAILED ceph_assert(clone_size.count(clone))
by Jake Grimmett 23 Mar '20

23 Mar '20

Dear All, We are having problems with a critical osd crashing on a Nautilus (14.2.8) cluster. This is a critical failure, as the osd is part of a pg that is otherwise "down+remapped" due to other osd's crashing; we were hoping the pg was going to repair itself, as there are plenty of free osds, but for some reason this pg never managed to get out of an undersized state. The osd starts OK, runs for a few minutes, then crashes with an assert, immediately after trying to backfill the pg that is "down+remapped" -7> 2020-03-23 15:28:15.368 7f15aeea8700 5 osd.287 pg_epoch: 35531 pg[5.750s2( v 35398'3381328 (35288'3378238,35398'3381328] local-lis/les=35530/35531 n=190408 ec=1821/1818 lis/c 35530/22903 les/c/f 35531/22917/0 35486/35530/35530) [234,354,304,388,125,25,427,226,77,154]/[2147483647,2147483647,287,388,125,25,427,226,77,154]p287(2) backfill=[234(0),304(2),354(1)] r=2 lpr=35530 pi=[22903,35530)/9 rops=1 crt=35398'3381328 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfilling mbc={} trimq=112 ps=121] backfill_pos is 5:0ae00653:::1000e49a8c6.000000d3:head -6> 2020-03-23 15:28:15.381 7f15cc9ec700 10 monclient: get_auth_request con 0x555b2f229800 auth_method 0 -5> 2020-03-23 15:28:15.381 7f15b86bb700 2 osd.287 35531 ms_handle_reset con 0x555b2fef7400 session 0x555b2f363600 -4> 2020-03-23 15:28:15.391 7f15c04c5700 5 prioritycache tune_memory target: 4294967296 mapped: 805339136 unmapped: 1032192 heap: 806371328 old mem: 2845415832 new mem: 2845415832 -3> 2020-03-23 15:28:15.420 7f15cc9ec700 10 monclient: get_auth_request con 0x555b2fef7800 auth_method 0 -2> 2020-03-23 15:28:15.420 7f15b86bb700 2 osd.287 35531 ms_handle_reset con 0x555b2fef7c00 session 0x555b2f363c00 -1> 2020-03-23 15:28:15.476 7f15aeea8700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7f15aeea8700 time 2020-03-23 15:28:15.470166 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/osd/osd_types.cc: 5443: FAILED ceph_assert(clone_size.count(clone)) osd log (127KB) is here: <https://www.mrc-lmb.cam.ac.uk/scicomp/ceph-osd.287.log.gz> /var/log/ceph/ceph-osd.287.log.gz when the osd was running, the pg state is as follows [root@ceph7 ~]# ceph pg dump | grep ^5.750 5.750 190408 0 804 190119 0 569643615603 0 0 3090 3090 active+undersized+degraded+remapped+backfill_wait 2020-03-23 14:37:57.582509 35398'3381328 35491:3265627 [234,354,304,388,125,25,427,226,77,154] 234 [NONE,NONE,287,388,125,25,427,226,77,154] 287 24471'3200829 2020-01-28 15:48:35.574934 24471'3200829 2020-01-28 15:48:35.574934 112 with the osd down: [root@ceph7 ~]# ceph pg dump | grep ^5.750 dumped all 5.750 190408 0 0 0 0 569643615603 0 0 3090 3090 down+remapped 2020-03-23 15:28:28.345176 35398'3381328 35532:3265613 [234,354,304,388,125,25,427,226,77,154] 234 [NONE,NONE,NONE,388,125,25,427,226,77,154] 388 24471'3200829 2020-01-28 15:48:35.574934 24471'3200829 2020-01-28 15:48:35.574934 This cluster is being used to backup a live cephfs cluster and has 1.8PB of data, including 30 days of snapshots. We are using 8+2 EC. Any help appreciated, Jake Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

1 0

can't get healthy cluster to trim osdmaps (13.2.8)
by Nikola Ciprich 23 Mar '20

23 Mar '20

Hello dear cephers, lately, there's been some discussion about slow requests hanging in "wait for new map" status. At least in my case, it's being caused by osdmaps not being properly trimmed. I tried all possible steps to force osdmap pruning (restarting mons, restarting everyging, poking crushmap), to no avail. Still all OSDs keep min osdmap version 1, while newest is 4734. Otherwise cluster is healthy, with no down OSDs, network communication works flawlessly, all seems to be fine. Just can't get old osdmaps to go away.. I's very small cluster and I've moved all production traffic elsewhere, so I'm free to investigate and debug, however I'm out of ideas on what to try or where to look. Any ideas somebody please? The cluster is running 13.2.8 I'd be very grateful for any tips with best regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

2 4

How to recover/mount mirrored rbd image for file recovery
by Ml Ml 23 Mar '20

23 Mar '20

Hello, my goal is to back up a proxmox cluster with rbd-mirror for desaster recovery. Promoting/Demoting, etc.. works great. But how can i access a single file on the mirrored cluster? I tried: root@ceph01:~# rbd-nbd --read-only map cluster5-rbd/vm-114-disk-1 --cluster backup /dev/nbd1 But i get: root@ceph01:~# fdisk -l /dev/nbd1 fdisk: cannot open /dev/nbd1: Input/output error dmesg shows stuff like: [Thu Mar 19 09:29:55 2020] nbd1: unable to read partition table [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) Here is my state: root@ceph01:~# rbd --cluster backup mirror pool status cluster5-rbd --verbose health: OK images: 3 total 3 replaying vm-106-disk-0: global_id: 0bc18ee1-1749-4787-a45d-01c7e946ff06 state: up+replaying description: replaying, master_position=[object_number=3, tag_tid=2, entry_tid=3], mirror_position=[object_number=3, tag_tid=2, entry_tid=3], entries_behind_master=0 last_update: 2020-03-19 09:29:17 vm-114-disk-1: global_id: 2219ffa9-a4e0-4f89-b352-ff30b1ffe9b9 state: up+replaying description: replaying, master_position=[object_number=390, tag_tid=6, entry_tid=334290], mirror_position=[object_number=382, tag_tid=6, entry_tid=328526], entries_behind_master=5764 last_update: 2020-03-19 09:29:17 vm-115-disk-0: global_id: 2b0af493-14c1-4b10-b557-84928dc37dd1 state: up+replaying description: replaying, master_position=[object_number=72, tag_tid=1, entry_tid=67796], mirror_position=[object_number=72, tag_tid=1, entry_tid=67796], entries_behind_master=0 last_update: 2020-03-19 09:29:17 More dmesg stuff: [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:29:55 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: 95 callbacks suppressed [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 0 [Thu Mar 19 09:30:02 2020] buffer_io_error: 94 callbacks suppressed [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 0, async page read [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 1 [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 1, async page read [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 2 [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 2, async page read [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 3 [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 3, async page read [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 4 [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 4, async page read [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 5 [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 5, async page read [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 6 [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 6, async page read [Thu Mar 19 09:30:02 2020] block nbd1: Other side returned error (30) [Thu Mar 19 09:30:02 2020] blk_update_request: I/O error, dev nbd1, sector 7 [Thu Mar 19 09:30:02 2020] Buffer I/O error on dev nbd1, logical block 7, async page read Do i have to stop the replaying or how can i mount the image on the backup cluster? Thanks, Michael

3 6

Re: [15.1.1-rc] - "Module 'dashboard' has failed: ('pwdUpdateRequired',)"
by Gencer W. Genç 23 Mar '20

23 Mar '20

Hi Lenz, Yeah, I saw the PR and i still hit the issue today. In the meantime while @Volker investigates, is there a workaround to bring dashboard back? Or should I wait for @Volker's investigation? P.S.: I fopund this too: https://tracker.ceph.com/issues/44271 Thanks, Gencer. ======= REPLY ABOVE ======= Hi Gencer, On 2020-03-22 09:37, Gencer W. Genç wrote: ... Hmm, I thought we had fixed that bug by merging the following fix: https://github.com/ceph/ceph/pull/33513 @Volker, would you mind taking a look at this? Thanks in advance! Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

2 1

How can I recover PGs in state 'unknown', where OSD location seems to be lost?
by Mark S. Holliman 23 Mar '20

23 Mar '20

Hi all, I have a large distributed ceph cluster that recently broke with all PGs housed at a single site getting marked as 'unknown' after a run of the Ceph Ansible playbook (which was being used to expand the cluster at a third site). Is there a way to recover the location of PGs in this state, or a way to fall back to a previous config where things were working? Or a way to scan the OSDs to determine which PGs are housed there? All the OSDs are still in place and reporting as healthy, it's just the PG locations that are missing. For info: the ceph cluster is used to provide a single shared CephFS mount for a distributed batch cluster, and it includes workers and pools of OSDs from three different OpenStack clouds. Ceph version: 13.2.8 Here is the system health: [root@euclid-edi-ctrl-0 ~]# ceph -s cluster: id: 0fe7e967-ecd6-46d4-9f6b-224539073d3b health: HEALTH_WARN insufficient standby MDS daemons available 1 MDSs report slow metadata IOs Reduced data availability: 1024 pgs inactive 6 slow ops, oldest one blocked for 244669 sec, mon.euclid-edi-ctrl-0 has slow ops too few PGs per OSD (26 < min 30) services: mon: 4 daemons, quorum euclid-edi-ctrl-0,euclid-cam-proxy-0,euclid-imp-proxy-0,euclid-ral-proxy-0 mgr: euclid-edi-ctrl-0(active), standbys: euclid-imp-proxy-0, euclid-cam-proxy-0, euclid-ral-proxy-0 mds: cephfs-2/2/2 up {0=euclid-ral-proxy-0=up:active,1=euclid-cam-proxy-0=up:active} osd: 269 osds: 269 up, 269 in data: pools: 5 pools, 5120 pgs objects: 30.54 M objects, 771 GiB usage: 3.8 TiB used, 41 TiB / 45 TiB avail pgs: 20.000% pgs unknown 4095 active+clean 1024 unknown 1 active+clean+scrubbing OSD Pools: [root@euclid-edi-ctrl-0 ~]# ceph osd lspools 1 cephfs_data 2 cephfs_metadata 3 euclid_cam 4 euclid_ral 5 euclid_imp [root@euclid-edi-ctrl-0 ~]# ceph pg dump_pools_json dumped pools POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG 5 0 0 0 0 0 0 0 0 0 0 1 16975540 0 0 0 0 79165311663 0 0 6243475 6243475 2 5171099 0 0 0 0 551991405 126879876 270829 3122183 3122183 3 8393436 0 0 0 0 748466429315 0 0 1556647 1556647 4 0 0 0 0 0 0 0 0 0 0 [root@euclid-edi-ctrl-0 ~]# ceph health detail ... PG_AVAILABILITY Reduced data availability: 1024 pgs inactive pg 4.3c8 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3ca is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3cb is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d0 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d1 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d2 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d3 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d4 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d5 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d6 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d7 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d8 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3d9 is stuck inactive for 246794.767182, current state unknown, last acting [] pg 4.3da is stuck inactive for 246794.767182, current state unknown, last acting [] ... [root@euclid-edi-ctrl-0 ~]# ceph pg map 4.3c8 osdmap e284992 pg 4.3c8 (4.3c8) -> up [] acting [] Cheers, Mark

1 0

Ceph pool quotas
by Stolte, Felix 23 Mar '20

23 Mar '20

Hey guys, a short question about pool quotas. Do they apply to stats attributes “stored” or “bytes_used” (Is replication counted for or not)? Regards Felix IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

3 2

Newbie to Ceph jacked up his monitor
by Jarett DeAngelis 22 Mar '20

22 Mar '20

hi folks. I seem to have gotten myself into a situation where my monitor directory filled itself up entirely while I wasn't looking, and now my monitor is unresponsive. (this is in a 3-node cluster). it grew a whole lot right after I added a number of OSDs and it started rebalancing. how can I get my monitor back into a working state so the rebalance can finish and the system becomes usable again? thx!

1 0

Maximum limit of lifecycle rule length
by Amit Ghadge 22 Mar '20

22 Mar '20

He All, We set rgw_lc_max_rules to 10000 but we seen the issue while the xml rule length are > 1MB It return InvalidRange, the format is below, <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <Status>Enabled</Status><Prefix>test_1/</Prefix><Expiration><Days>1</Days></Expiration>.</Rule> . . . </LifecycleConfiguration> Any reason why Ceph not allowed lc rule length > 1MB. Thanks, AmitG

1 0

Re: Cephfs mount error 1 = Operation not permitted
by Eugen Block 22 Mar '20

22 Mar '20

Hi, have you tried to mount with the secret only instead of a secret file? mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=<SECRET> If that works your secret file is not right. If not you should check if the client actually has access to the cephfs pools ('ceph auth list'). Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > I am still very new to ceph and I have just set up my first small > test cluster. I have Cephfs enabled (named cephfs) and everything is > good in the dashboard. I added an authorized user key for cephfs with: > > ceph fs authorize cephfs client.1 / r / rw > > I then copied the key to a file with: > > ceph auth get-key client.1 > /tmp/client.1.secret > > Copied the file over to the client and then attempt mount witth the > kernel driver: > > mount -t ceph ceph-n4:6789:/ /ceph -o > name=client.1,secretfile=/root/client.1.secret > mount error 1 = Operation not permitted > > I looked in the logs on the mds (which is also the mgr and mon for > the cluster) and I don't see any events logged for this. I also > tried the mount command with verbose and I didn't get any further > detail. Any tips would be most appreciated. > > -- > > Scott Dungan > California Institute of Technology > Office: (626) 395-3170 > sdungan(a)caltech.edu<mailto:sdungan@caltech.edu> > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Cephfs mount error 1 = Operation not permitted
by Dungan, Scott A. 21 Mar '20

21 Mar '20

I am still very new to ceph and I have just set up my first small test cluster. I have Cephfs enabled (named cephfs) and everything is good in the dashboard. I added an authorized user key for cephfs with: ceph fs authorize cephfs client.1 / r / rw I then copied the key to a file with: ceph auth get-key client.1 > /tmp/client.1.secret Copied the file over to the client and then attempt mount witth the kernel driver: mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secretfile=/root/client.1.secret mount error 1 = Operation not permitted I looked in the logs on the mds (which is also the mgr and mon for the cluster) and I don't see any events logged for this. I also tried the mount command with verbose and I didn't get any further detail. Any tips would be most appreciated. -- Scott Dungan California Institute of Technology Office: (626) 395-3170 sdungan(a)caltech.edu<mailto:sdungan@caltech.edu>

1 0

Re: Questions on Ceph cluster without OS disks
by Marc Roos 21 Mar '20

21 Mar '20

I would say it is not a 'proven technology' otherwise you would see a wide spread implementation and adaptation of this method. However if you really need the physical disk space, it is a solution. Although I also would have questions on creating an extra redundant environment to service remote booting, just to spare a os disk position. Maybe this makes more sence in really big environments. -----Original Message----- From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] Sent: 21 March 2020 13:54 To: Martin Verges; ceph-users Subject: [ceph-users] Questions on Ceph cluster without OS disks Hello， Martin， I notice that Croit advocate the use of ceph cluster without OS disks, but with PXE boot. Do you use a NFS server to serve the root file system for each node? such as hosting configuration files, user and password, log files, etc. My question is, will the NFS server be a single point of failure? If the NFS server goes down, the network experience any outage, ceph nodes may not be able to write to the local file systems, possibly leading to service outage. How do you deal with the above potential issues in production? I am a bit worried... best regards, samuel

2 1

New 3 node Ceph cluster
by Amudhan P 21 Mar '20

21 Mar '20

Hi, I am planning to create a new 3 node ceph storage cluster. I will be using Cephfs + with samba for max 10 clients for upload and download. Storage Node HW is Intel Xeon E5v2 8 core single Proc, 32GB RAM and 10Gb Nic 2 nos., 6TB SATA HDD 24 Nos. each node, OS separate SSD disk. Earlier I have tested orchestration using ceph-deploy in the test setup. now, is there any other alternative to ceph-deploy? Can I restrict folder access to the user using cephfs + vfs samba or should I use ceph client + samba? Ubuntu or Centos? Any block size consideration for object size, metadata when using cephfs? Idea or suggestion from existing users. I am also going to start to explore all the above. regards Amudhan

6 8

ceph objecy storage client gui
by Ignazio Cassano 21 Mar '20

21 Mar '20

Hello All, I am looking for object storage freee/opensource client gui (linux and windows) for end users . I tried swiftstack but it is only for personal use. Help, please ? Ignazio

2 2

Docs@RSS
by Guilherme Geronimo 20 Mar '20

20 Mar '20

Hey guys, do you know who takes care of the docs.ceph.com? I mean, the tool, not the content. I would suggest to enable its FEED (RSS/ATOM). That would facilitate a lot of things... -- []'s Arthur (aKa Guilherme Geronimo)

1 0

crush rule question
by Philippe Van Hecke 20 Mar '20

20 Mar '20

Hi , i have a question regarding crush rules. I try to have the following: i have domain faillure on rack. currently we have 2 racks what i want to make is a crush rules for 3 replicas where at least 2 replicas are put on 2 differents rack and the third one can be on any of the 2. Is this possible and if yes how to write the crrush rule. Kr Philippe Van Hecke.

1 0

Full OSD's on cephfs_metadata pool
by Robert Ruge 19 Mar '20

19 Mar '20

Hi All. Nautilus 14.2.8. I came in this morning to find that six of my eight NVME OSD's that were housing the cephfs_metadata pool had mysteriously filled up and crashed overnight and they won't come back up. These OSD's are all single logical volume devices with no separate WAL or DB. I have tried extending the LV of one of the OSD's but it can't make use of it and I have added a separate db volume but that didn't help. In the meantime I have told the cluster to move cephfs_metadata back to HDD which it has kindly done and emptied my two live OSD's but I am left with 10 pgs inactive. BLUEFS_SPILLOVER BlueFS spillover detected on 6 OSD(s) osd.93 spilled over 521 MiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.95 spilled over 456 MiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.100 spilled over 2.1 GiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.107 spilled over 782 MiB metadata from 'db' device (26 GiB used of 50 GiB) to slow device osd.112 spilled over 1.3 GiB metadata from 'db' device (27 GiB used of 50 GiB) to slow device osd.115 spilled over 1.4 GiB metadata from 'db' device (27 GiB used of 50 GiB) to slow device PG_AVAILABILITY Reduced data availability: 10 pgs inactive, 10 pgs down pg 2.4e is down, acting [60,6,120] pg 2.60 is down, acting [105,132,15] pg 2.61 is down, acting [8,13,112] pg 2.72 is down, acting [93,112,0] pg 2.9f is down, acting [117,1,35] pg 2.b9 is down, acting [95,25,6] pg 2.c3 is down, acting [97,139,5] pg 2.c6 is down, acting [95,7,127] pg 2.d1 is down, acting [36,107,17] pg 2.f4 is down, acting [23,117,138] Can I backup and recreate an OSD on a larger volume? Can I remove a good pg from an offline OSD to remove some space? Ceph-bluestore-tool repair fails. "bluefs enospc" seems to be the critical error. So currently my cephfs is unavailable so any help would be greatly appreciated. Regards Robert Ruge Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

4 5

Re: OSDs continuously restarting under load
by Igor Fedotov 19 Mar '20

19 Mar '20

Hi, Samuel, I've never seen that sort of signal in the real life: 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** I suppose this has some hardware roots. Have you checked dmesg output? Just in case, here is some info on "Bus Error" signal, may be it will provide some insight: https://en.wikipedia.org/wiki/Bus_error Thanks, Igor On 3/18/2020 5:06 PM, huxiaoyu(a)horebdata.cn wrote: > Hello， folks, > > I am trying to add a ceph node into an existing ceph cluster. Once the reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd becomes unresponsive and restarting, eventually go down. > > What could be the problem? Any suggestion would be highly appreciated. > > best regards, > > samuel > > **************************************************** > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 12 hybrid 1.00000 1.00000 3.81TiB 38.3GiB 3.77TiB 0.98 1.32 316 > 13 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 308 > 14 hybrid 1.00000 1.00000 3.81TiB 36.9GiB 3.77TiB 0.95 1.27 301 > 15 hybrid 1.00000 1.00000 3.81TiB 37.1GiB 3.77TiB 0.95 1.28 297 > 0 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 305 > 1 hybrid 1.00000 1.00000 3.81TiB 38.2GiB 3.77TiB 0.98 1.31 309 > 2 hybrid 1.00000 1.00000 3.81TiB 37.4GiB 3.77TiB 0.96 1.29 296 > 3 hybrid 1.00000 1.00000 3.81TiB 37.9GiB 3.77TiB 0.97 1.30 303 > 4 hdd 0.20000 1.00000 3.42TiB 10.5GiB 3.41TiB 0.30 0.40 0 > 5 hdd 0.20000 1.00000 3.42TiB 9.63GiB 3.41TiB 0.28 0.37 87 > 6 hdd 0.20000 1.00000 3.42TiB 1.91GiB 3.42TiB 0.05 0.07 0 > 7 hdd 0.20000 1.00000 3.42TiB 11.3GiB 3.41TiB 0.32 0.43 83 > 16 hdd 0.39999 1.00000 1.79TiB 16.3GiB 1.78TiB 0.89 1.19 142 > TOTAL 45.9TiB 351GiB 45.6TiB 0.75 > > ------------------------------------------------------------------------------------ 日志 > > root@node81:/var/log/ceph# cat ceph-osd.6.log | grep load_pgs > 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > -1324> 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > -1323> 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs > -466> 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > -465> 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs > 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs > -422> 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > -421> 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs > 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > > > 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none > 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none > 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete > 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > in thread 201e35fdb40 thread_name:tp_osd_tp > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) > 1: (()+0x145882c) [0x2000245882c] > 2: (()+0x19890) [0x2000c54b890] > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] > 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] > 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > 19: (()+0x80fc) [0x2000c53a0fc] > 20: (()+0x119854) [0x2000f2ad854] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- begin dump of recent events --- > -147> 2020-03-18 18:37:51.039443 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_dump hook 0x2000ce09e40 > -146> 2020-03-18 18:37:51.039716 2000f374000 5 asok(0x2000cd7f230) register_command 1 hook 0x2000ce09e40 > -145> 2020-03-18 18:37:51.039736 2000f374000 5 asok(0x2000cd7f230) register_command perf dump hook 0x2000ce09e40 > -144> 2020-03-18 18:37:51.039769 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_schema hook 0x2000ce09e40 > -143> 2020-03-18 18:37:51.039789 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram dump hook 0x2000ce09e40 > -142> 2020-03-18 18:37:51.039807 2000f374000 5 asok(0x2000cd7f230) register_command 2 hook 0x2000ce09e40 > -141> 2020-03-18 18:37:51.039823 2000f374000 5 asok(0x2000cd7f230) register_command perf schema hook 0x2000ce09e40 > -140> 2020-03-18 18:37:51.039843 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram schema hook 0x2000ce09e40 > -139> 2020-03-18 18:37:51.039863 2000f374000 5 asok(0x2000cd7f230) register_command perf reset hook 0x2000ce09e40 > -138> 2020-03-18 18:37:51.039881 2000f374000 5 asok(0x2000cd7f230) register_command config show hook 0x2000ce09e40 > -137> 2020-03-18 18:37:51.039899 2000f374000 5 asok(0x2000cd7f230) register_command config help hook 0x2000ce09e40 > -136> 2020-03-18 18:37:51.039928 2000f374000 5 asok(0x2000cd7f230) register_command config set hook 0x2000ce09e40 > -135> 2020-03-18 18:37:51.039949 2000f374000 5 asok(0x2000cd7f230) register_command config get hook 0x2000ce09e40 > -134> 2020-03-18 18:37:51.039967 2000f374000 5 asok(0x2000cd7f230) register_command config diff hook 0x2000ce09e40 > -133> 2020-03-18 18:37:51.039985 2000f374000 5 asok(0x2000cd7f230) register_command config diff get hook 0x2000ce09e40 > -132> 2020-03-18 18:37:51.040005 2000f374000 5 asok(0x2000cd7f230) register_command log flush hook 0x2000ce09e40 > -131> 2020-03-18 18:37:51.040021 2000f374000 5 asok(0x2000cd7f230) register_command log dump hook 0x2000ce09e40 > -130> 2020-03-18 18:37:51.040038 2000f374000 5 asok(0x2000cd7f230) register_command log reopen hook 0x2000ce09e40 > -129> 2020-03-18 18:37:51.040189 2000f374000 5 asok(0x2000cd7f230) register_command dump_mempools hook 0x2000ce0d038 > -128> 2020-03-18 18:37:51.099580 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -127> 2020-03-18 18:37:51.102046 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -126> 2020-03-18 18:37:51.102148 2000f374000 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 1159687 > -125> 2020-03-18 18:37:51.109860 2000f374000 0 pidfile_write: ignore empty --pid-file > -124> 2020-03-18 18:37:51.130256 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -123> 2020-03-18 18:37:51.183798 2000f374000 0 load: jerasure load: lrc > -122> 2020-03-18 18:37:51.184330 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -121> 2020-03-18 18:37:51.184492 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -120> 2020-03-18 18:37:51.184741 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -119> 2020-03-18 18:37:51.185841 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -118> 2020-03-18 18:37:51.186232 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) close > -117> 2020-03-18 18:37:51.498718 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -116> 2020-03-18 18:37:51.498748 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -115> 2020-03-18 18:37:51.498811 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -114> 2020-03-18 18:37:51.499234 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -113> 2020-03-18 18:37:51.500074 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block.db type kernel > -112> 2020-03-18 18:37:51.500096 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open path /var/lib/ceph/osd/ceph-6/block.db > -111> 2020-03-18 18:37:51.500170 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -110> 2020-03-18 18:37:51.500815 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, 38146 MB) block_size 4096 (4096 B) rotational > -109> 2020-03-18 18:37:51.502625 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -108> 2020-03-18 18:37:51.502651 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -107> 2020-03-18 18:37:51.502718 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -106> 2020-03-18 18:37:51.503137 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -105> 2020-03-18 18:37:51.549269 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB > -104> 2020-03-18 18:37:51.549349 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel > -103> 2020-03-18 18:37:51.552610 2000f374000 0 set rocksdb option compaction_threads = 32 > -102> 2020-03-18 18:37:51.552652 2000f374000 0 set rocksdb option compression = kNoCompression > -101> 2020-03-18 18:37:51.553442 2000f374000 0 set rocksdb option flusher_threads = 8 > -100> 2020-03-18 18:37:51.553508 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 > -99> 2020-03-18 18:37:51.553536 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 > -98> 2020-03-18 18:37:51.553559 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 > -97> 2020-03-18 18:37:51.553579 2000f374000 0 set rocksdb option max_background_compactions = 64 > -96> 2020-03-18 18:37:51.553601 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB > -95> 2020-03-18 18:37:51.553624 2000f374000 0 set rocksdb option max_write_buffer_number = 64 > -94> 2020-03-18 18:37:51.553646 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 > -93> 2020-03-18 18:37:51.553665 2000f374000 0 set rocksdb option recycle_log_file_num = 64 > -92> 2020-03-18 18:37:51.553687 2000f374000 0 set rocksdb option target_file_size_base = 4MB > -91> 2020-03-18 18:37:51.553708 2000f374000 0 set rocksdb option write_buffer_size = 4MB > -90> 2020-03-18 18:37:51.553892 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB > -89> 2020-03-18 18:37:51.553923 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel > -88> 2020-03-18 18:37:51.553948 2000f374000 0 set rocksdb option compaction_threads = 32 > -87> 2020-03-18 18:37:51.553973 2000f374000 0 set rocksdb option compression = kNoCompression > -86> 2020-03-18 18:37:51.553994 2000f374000 0 set rocksdb option flusher_threads = 8 > -85> 2020-03-18 18:37:51.554016 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 > -84> 2020-03-18 18:37:51.554043 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 > -83> 2020-03-18 18:37:51.554065 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 > -82> 2020-03-18 18:37:51.554084 2000f374000 0 set rocksdb option max_background_compactions = 64 > -81> 2020-03-18 18:37:51.554106 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB > -80> 2020-03-18 18:37:51.554133 2000f374000 0 set rocksdb option max_write_buffer_number = 64 > -79> 2020-03-18 18:37:51.554154 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 > -78> 2020-03-18 18:37:51.554174 2000f374000 0 set rocksdb option recycle_log_file_num = 64 > -77> 2020-03-18 18:37:51.554196 2000f374000 0 set rocksdb option target_file_size_base = 4MB > -76> 2020-03-18 18:37:51.554232 2000f374000 0 set rocksdb option write_buffer_size = 4MB > -75> 2020-03-18 18:37:56.382110 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello > -74> 2020-03-18 18:37:56.383845 2000f374000 0 _get_class not permitted to load lua > -73> 2020-03-18 18:37:56.386594 2000f374000 0 _get_class not permitted to load sdk > -72> 2020-03-18 18:37:56.395800 2000f374000 0 _get_class not permitted to load kvs > -71> 2020-03-18 18:37:56.398226 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs > -70> 2020-03-18 18:37:56.433293 2000f374000 0 osd.6 5247 crush map has features 432629239337189376, adjusting msgr requires for clients > -69> 2020-03-18 18:37:56.433330 2000f374000 0 osd.6 5247 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons > -68> 2020-03-18 18:37:56.433357 2000f374000 0 osd.6 5247 crush map has features 1009089991640629248, adjusting msgr requires for osds > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > -65> 2020-03-18 18:39:03.377040 2000f374000 0 osd.6 5247 using weightedpriority op queue with priority op cut off at 196. > -64> 2020-03-18 18:39:03.413901 2000f374000 -1 osd.6 5247 log_to_monitors {default=true} > -63> 2020-03-18 18:39:03.663128 2000f374000 0 osd.6 5247 done with init, starting boot process > -62> 2020-03-18 18:39:03.663856 201d65fdb40 4 mgrc handle_mgr_map Got map version 34 > -61> 2020-03-18 18:39:03.663947 201d65fdb40 4 mgrc handle_mgr_map Active mgr is now 192.168.230.120:6808/44007 > -60> 2020-03-18 18:39:03.663972 201d65fdb40 4 mgrc reconnect Starting new session with 192.168.230.120:6808/44007 > -59> 2020-03-18 18:39:03.667814 201d65fdb40 4 mgrc handle_mgr_configure stats_period=5 > -58> 2020-03-18 18:39:03.667831 201d65fdb40 4 mgrc handle_mgr_configure updated stats threshold: 5 > -57> 2020-03-18 18:39:03.752253 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -56> 2020-03-18 18:39:03.754040 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -55> 2020-03-18 18:39:03.762441 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -54> 2020-03-18 18:39:03.762860 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -53> 2020-03-18 18:39:03.765775 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -52> 2020-03-18 18:39:03.767284 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -51> 2020-03-18 18:39:03.769434 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -50> 2020-03-18 18:39:03.775190 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -49> 2020-03-18 18:39:04.009755 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -48> 2020-03-18 18:39:05.177544 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -47> 2020-03-18 18:39:05.402465 2000dae5b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -46> 2020-03-18 18:39:09.483237 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 flags = none tried to add 3097'452 flags = none > -45> 2020-03-18 18:39:09.483318 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 flags = none tried to add 1915'212 flags = none > -44> 2020-03-18 18:39:09.483336 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 flags = none tried to add 1915'213 flags = none > -43> 2020-03-18 18:39:09.483351 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 flags = none tried to add 5165'734 flags = none > -42> 2020-03-18 18:39:09.483366 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 flags = none tried to add 2855'272 flags = none > -41> 2020-03-18 18:39:09.483381 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 flags = none tried to add 1915'214 flags = none > -40> 2020-03-18 18:39:09.483395 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 flags = none tried to add 3798'560 flags = none > -39> 2020-03-18 18:39:09.483409 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 flags = none tried to add 2197'242 flags = none > -38> 2020-03-18 18:39:09.483423 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 flags = delete tried to add 3099'543 flags = delete > -37> 2020-03-18 18:39:09.483438 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 flags = delete tried to add 5169'1101 flags = delete > -36> 2020-03-18 18:39:09.483454 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 flags = none tried to add 1915'215 flags = none > -35> 2020-03-18 18:39:09.483469 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 flags = none tried to add 5175'1280 flags = none > -34> 2020-03-18 18:39:09.483483 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 flags = delete tried to add 5166'844 flags = delete > -33> 2020-03-18 18:39:09.483497 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 flags = delete tried to add 1805'174 flags = delete > -32> 2020-03-18 18:39:09.483511 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 flags = none tried to add 1915'216 flags = none > -31> 2020-03-18 18:39:09.483525 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 flags = none tried to add 5175'1283 flags = none > -30> 2020-03-18 18:39:09.483539 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 flags = none tried to add 1915'217 flags = none > -29> 2020-03-18 18:39:09.483553 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 flags = none tried to add 5175'1284 flags = none > -28> 2020-03-18 18:39:09.483567 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 flags = delete tried to add 1805'179 flags = delete > -27> 2020-03-18 18:39:09.483582 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 flags = delete tried to add 5166'785 flags = delete > -26> 2020-03-18 18:39:09.483595 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 flags = delete tried to add 1805'200 flags = delete > -25> 2020-03-18 18:39:09.483610 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 flags = none tried to add 3894'562 flags = none > -24> 2020-03-18 18:39:09.483625 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 flags = none tried to add 5172'1141 flags = none > -23> 2020-03-18 18:39:09.483639 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 flags = none tried to add 1915'218 flags = none > -22> 2020-03-18 18:39:09.483654 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 flags = delete tried to add 1805'198 flags = delete > -21> 2020-03-18 18:39:09.483667 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 flags = delete tried to add 3908'593 flags = delete > -20> 2020-03-18 18:39:09.483681 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 flags = none tried to add 5176'1523 flags = none > -19> 2020-03-18 18:39:09.483695 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 flags = none tried to add 5165'735 flags = none > -18> 2020-03-18 18:39:09.483710 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 flags = none tried to add 3908'563 flags = none > -17> 2020-03-18 18:39:09.483724 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 flags = none tried to add 4427'730 flags = none > -16> 2020-03-18 18:39:09.483738 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 flags = none tried to add 4743'732 flags = none > -15> 2020-03-18 18:39:09.483752 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 flags = none tried to add 5175'1287 flags = none > -14> 2020-03-18 18:39:09.483766 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 flags = none tried to add 5172'1144 flags = none > -13> 2020-03-18 18:39:09.483782 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 flags = none tried to add 1915'219 flags = none > -12> 2020-03-18 18:39:09.483796 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 flags = delete tried to add 3092'344 flags = delete > -11> 2020-03-18 18:39:09.483811 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 flags = delete tried to add 3092'381 flags = delete > -10> 2020-03-18 18:39:09.483825 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 flags = delete tried to add 1805'197 flags = delete > -9> 2020-03-18 18:39:09.483840 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 flags = delete tried to add 1805'199 flags = delete > -8> 2020-03-18 18:39:09.483854 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 flags = none tried to add 2197'271 flags = none > -7> 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none > -6> 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none > -5> 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete > -4> 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -3> 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -2> 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -1> 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 0> 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > in thread 201e35fdb40 thread_name:tp_osd_tp > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) > 1: (()+0x145882c) [0x2000245882c] > 2: (()+0x19890) [0x2000c54b890] > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] > 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] > 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > 19: (()+0x80fc) [0x2000c53a0fc] > 20: (()+0x119854) [0x2000f2ad854] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

OSDs continuously restarting under load
by huxiaoyu＠horebdata.cn 18 Mar '20

18 Mar '20

Hello， folks, I am trying to add a ceph node into an existing ceph cluster. Once the reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd becomes unresponsive and restarting, eventually go down. What could be the problem? Any suggestion would be highly appreciated. best regards, samuel **************************************************** root@node81:/var/log/ceph# root@node81:/var/log/ceph# root@node81:/var/log/ceph# root@node81:/var/log/ceph# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 12 hybrid 1.00000 1.00000 3.81TiB 38.3GiB 3.77TiB 0.98 1.32 316 13 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 308 14 hybrid 1.00000 1.00000 3.81TiB 36.9GiB 3.77TiB 0.95 1.27 301 15 hybrid 1.00000 1.00000 3.81TiB 37.1GiB 3.77TiB 0.95 1.28 297 0 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 305 1 hybrid 1.00000 1.00000 3.81TiB 38.2GiB 3.77TiB 0.98 1.31 309 2 hybrid 1.00000 1.00000 3.81TiB 37.4GiB 3.77TiB 0.96 1.29 296 3 hybrid 1.00000 1.00000 3.81TiB 37.9GiB 3.77TiB 0.97 1.30 303 4 hdd 0.20000 1.00000 3.42TiB 10.5GiB 3.41TiB 0.30 0.40 0 5 hdd 0.20000 1.00000 3.42TiB 9.63GiB 3.41TiB 0.28 0.37 87 6 hdd 0.20000 1.00000 3.42TiB 1.91GiB 3.42TiB 0.05 0.07 0 7 hdd 0.20000 1.00000 3.42TiB 11.3GiB 3.41TiB 0.32 0.43 83 16 hdd 0.39999 1.00000 1.79TiB 16.3GiB 1.78TiB 0.89 1.19 142 TOTAL 45.9TiB 351GiB 45.6TiB 0.75 ------------------------------------------------------------------------------------ 日志 root@node81:/var/log/ceph# cat ceph-osd.6.log | grep load_pgs 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs -1324> 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs -1323> 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs -466> 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs -465> 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs -422> 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs -421> 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** in thread 201e35fdb40 thread_name:tp_osd_tp ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0x145882c) [0x2000245882c] 2: (()+0x19890) [0x2000c54b890] 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] 19: (()+0x80fc) [0x2000c53a0fc] 20: (()+0x119854) [0x2000f2ad854] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -147> 2020-03-18 18:37:51.039443 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_dump hook 0x2000ce09e40 -146> 2020-03-18 18:37:51.039716 2000f374000 5 asok(0x2000cd7f230) register_command 1 hook 0x2000ce09e40 -145> 2020-03-18 18:37:51.039736 2000f374000 5 asok(0x2000cd7f230) register_command perf dump hook 0x2000ce09e40 -144> 2020-03-18 18:37:51.039769 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_schema hook 0x2000ce09e40 -143> 2020-03-18 18:37:51.039789 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram dump hook 0x2000ce09e40 -142> 2020-03-18 18:37:51.039807 2000f374000 5 asok(0x2000cd7f230) register_command 2 hook 0x2000ce09e40 -141> 2020-03-18 18:37:51.039823 2000f374000 5 asok(0x2000cd7f230) register_command perf schema hook 0x2000ce09e40 -140> 2020-03-18 18:37:51.039843 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram schema hook 0x2000ce09e40 -139> 2020-03-18 18:37:51.039863 2000f374000 5 asok(0x2000cd7f230) register_command perf reset hook 0x2000ce09e40 -138> 2020-03-18 18:37:51.039881 2000f374000 5 asok(0x2000cd7f230) register_command config show hook 0x2000ce09e40 -137> 2020-03-18 18:37:51.039899 2000f374000 5 asok(0x2000cd7f230) register_command config help hook 0x2000ce09e40 -136> 2020-03-18 18:37:51.039928 2000f374000 5 asok(0x2000cd7f230) register_command config set hook 0x2000ce09e40 -135> 2020-03-18 18:37:51.039949 2000f374000 5 asok(0x2000cd7f230) register_command config get hook 0x2000ce09e40 -134> 2020-03-18 18:37:51.039967 2000f374000 5 asok(0x2000cd7f230) register_command config diff hook 0x2000ce09e40 -133> 2020-03-18 18:37:51.039985 2000f374000 5 asok(0x2000cd7f230) register_command config diff get hook 0x2000ce09e40 -132> 2020-03-18 18:37:51.040005 2000f374000 5 asok(0x2000cd7f230) register_command log flush hook 0x2000ce09e40 -131> 2020-03-18 18:37:51.040021 2000f374000 5 asok(0x2000cd7f230) register_command log dump hook 0x2000ce09e40 -130> 2020-03-18 18:37:51.040038 2000f374000 5 asok(0x2000cd7f230) register_command log reopen hook 0x2000ce09e40 -129> 2020-03-18 18:37:51.040189 2000f374000 5 asok(0x2000cd7f230) register_command dump_mempools hook 0x2000ce0d038 -128> 2020-03-18 18:37:51.099580 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb -127> 2020-03-18 18:37:51.102046 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb -126> 2020-03-18 18:37:51.102148 2000f374000 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 1159687 -125> 2020-03-18 18:37:51.109860 2000f374000 0 pidfile_write: ignore empty --pid-file -124> 2020-03-18 18:37:51.130256 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb -123> 2020-03-18 18:37:51.183798 2000f374000 0 load: jerasure load: lrc -122> 2020-03-18 18:37:51.184330 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel -121> 2020-03-18 18:37:51.184492 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block -120> 2020-03-18 18:37:51.184741 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -119> 2020-03-18 18:37:51.185841 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational -118> 2020-03-18 18:37:51.186232 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) close -117> 2020-03-18 18:37:51.498718 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel -116> 2020-03-18 18:37:51.498748 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block -115> 2020-03-18 18:37:51.498811 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -114> 2020-03-18 18:37:51.499234 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational -113> 2020-03-18 18:37:51.500074 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block.db type kernel -112> 2020-03-18 18:37:51.500096 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open path /var/lib/ceph/osd/ceph-6/block.db -111> 2020-03-18 18:37:51.500170 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -110> 2020-03-18 18:37:51.500815 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, 38146 MB) block_size 4096 (4096 B) rotational -109> 2020-03-18 18:37:51.502625 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel -108> 2020-03-18 18:37:51.502651 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block -107> 2020-03-18 18:37:51.502718 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway -106> 2020-03-18 18:37:51.503137 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational -105> 2020-03-18 18:37:51.549269 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB -104> 2020-03-18 18:37:51.549349 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel -103> 2020-03-18 18:37:51.552610 2000f374000 0 set rocksdb option compaction_threads = 32 -102> 2020-03-18 18:37:51.552652 2000f374000 0 set rocksdb option compression = kNoCompression -101> 2020-03-18 18:37:51.553442 2000f374000 0 set rocksdb option flusher_threads = 8 -100> 2020-03-18 18:37:51.553508 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 -99> 2020-03-18 18:37:51.553536 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 -98> 2020-03-18 18:37:51.553559 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 -97> 2020-03-18 18:37:51.553579 2000f374000 0 set rocksdb option max_background_compactions = 64 -96> 2020-03-18 18:37:51.553601 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB -95> 2020-03-18 18:37:51.553624 2000f374000 0 set rocksdb option max_write_buffer_number = 64 -94> 2020-03-18 18:37:51.553646 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 -93> 2020-03-18 18:37:51.553665 2000f374000 0 set rocksdb option recycle_log_file_num = 64 -92> 2020-03-18 18:37:51.553687 2000f374000 0 set rocksdb option target_file_size_base = 4MB -91> 2020-03-18 18:37:51.553708 2000f374000 0 set rocksdb option write_buffer_size = 4MB -90> 2020-03-18 18:37:51.553892 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB -89> 2020-03-18 18:37:51.553923 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel -88> 2020-03-18 18:37:51.553948 2000f374000 0 set rocksdb option compaction_threads = 32 -87> 2020-03-18 18:37:51.553973 2000f374000 0 set rocksdb option compression = kNoCompression -86> 2020-03-18 18:37:51.553994 2000f374000 0 set rocksdb option flusher_threads = 8 -85> 2020-03-18 18:37:51.554016 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 -84> 2020-03-18 18:37:51.554043 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 -83> 2020-03-18 18:37:51.554065 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 -82> 2020-03-18 18:37:51.554084 2000f374000 0 set rocksdb option max_background_compactions = 64 -81> 2020-03-18 18:37:51.554106 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB -80> 2020-03-18 18:37:51.554133 2000f374000 0 set rocksdb option max_write_buffer_number = 64 -79> 2020-03-18 18:37:51.554154 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 -78> 2020-03-18 18:37:51.554174 2000f374000 0 set rocksdb option recycle_log_file_num = 64 -77> 2020-03-18 18:37:51.554196 2000f374000 0 set rocksdb option target_file_size_base = 4MB -76> 2020-03-18 18:37:51.554232 2000f374000 0 set rocksdb option write_buffer_size = 4MB -75> 2020-03-18 18:37:56.382110 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello -74> 2020-03-18 18:37:56.383845 2000f374000 0 _get_class not permitted to load lua -73> 2020-03-18 18:37:56.386594 2000f374000 0 _get_class not permitted to load sdk -72> 2020-03-18 18:37:56.395800 2000f374000 0 _get_class not permitted to load kvs -71> 2020-03-18 18:37:56.398226 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -70> 2020-03-18 18:37:56.433293 2000f374000 0 osd.6 5247 crush map has features 432629239337189376, adjusting msgr requires for clients -69> 2020-03-18 18:37:56.433330 2000f374000 0 osd.6 5247 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons -68> 2020-03-18 18:37:56.433357 2000f374000 0 osd.6 5247 crush map has features 1009089991640629248, adjusting msgr requires for osds -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs -65> 2020-03-18 18:39:03.377040 2000f374000 0 osd.6 5247 using weightedpriority op queue with priority op cut off at 196. -64> 2020-03-18 18:39:03.413901 2000f374000 -1 osd.6 5247 log_to_monitors {default=true} -63> 2020-03-18 18:39:03.663128 2000f374000 0 osd.6 5247 done with init, starting boot process -62> 2020-03-18 18:39:03.663856 201d65fdb40 4 mgrc handle_mgr_map Got map version 34 -61> 2020-03-18 18:39:03.663947 201d65fdb40 4 mgrc handle_mgr_map Active mgr is now 192.168.230.120:6808/44007 -60> 2020-03-18 18:39:03.663972 201d65fdb40 4 mgrc reconnect Starting new session with 192.168.230.120:6808/44007 -59> 2020-03-18 18:39:03.667814 201d65fdb40 4 mgrc handle_mgr_configure stats_period=5 -58> 2020-03-18 18:39:03.667831 201d65fdb40 4 mgrc handle_mgr_configure updated stats threshold: 5 -57> 2020-03-18 18:39:03.752253 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -56> 2020-03-18 18:39:03.754040 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -55> 2020-03-18 18:39:03.762441 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -54> 2020-03-18 18:39:03.762860 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -53> 2020-03-18 18:39:03.765775 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -52> 2020-03-18 18:39:03.767284 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -51> 2020-03-18 18:39:03.769434 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -50> 2020-03-18 18:39:03.775190 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -49> 2020-03-18 18:39:04.009755 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer -48> 2020-03-18 18:39:05.177544 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -47> 2020-03-18 18:39:05.402465 2000dae5b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -46> 2020-03-18 18:39:09.483237 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 flags = none tried to add 3097'452 flags = none -45> 2020-03-18 18:39:09.483318 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 flags = none tried to add 1915'212 flags = none -44> 2020-03-18 18:39:09.483336 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 flags = none tried to add 1915'213 flags = none -43> 2020-03-18 18:39:09.483351 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 flags = none tried to add 5165'734 flags = none -42> 2020-03-18 18:39:09.483366 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 flags = none tried to add 2855'272 flags = none -41> 2020-03-18 18:39:09.483381 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 flags = none tried to add 1915'214 flags = none -40> 2020-03-18 18:39:09.483395 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 flags = none tried to add 3798'560 flags = none -39> 2020-03-18 18:39:09.483409 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 flags = none tried to add 2197'242 flags = none -38> 2020-03-18 18:39:09.483423 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 flags = delete tried to add 3099'543 flags = delete -37> 2020-03-18 18:39:09.483438 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 flags = delete tried to add 5169'1101 flags = delete -36> 2020-03-18 18:39:09.483454 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 flags = none tried to add 1915'215 flags = none -35> 2020-03-18 18:39:09.483469 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 flags = none tried to add 5175'1280 flags = none -34> 2020-03-18 18:39:09.483483 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 flags = delete tried to add 5166'844 flags = delete -33> 2020-03-18 18:39:09.483497 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 flags = delete tried to add 1805'174 flags = delete -32> 2020-03-18 18:39:09.483511 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 flags = none tried to add 1915'216 flags = none -31> 2020-03-18 18:39:09.483525 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 flags = none tried to add 5175'1283 flags = none -30> 2020-03-18 18:39:09.483539 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 flags = none tried to add 1915'217 flags = none -29> 2020-03-18 18:39:09.483553 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 flags = none tried to add 5175'1284 flags = none -28> 2020-03-18 18:39:09.483567 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 flags = delete tried to add 1805'179 flags = delete -27> 2020-03-18 18:39:09.483582 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 flags = delete tried to add 5166'785 flags = delete -26> 2020-03-18 18:39:09.483595 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 flags = delete tried to add 1805'200 flags = delete -25> 2020-03-18 18:39:09.483610 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 flags = none tried to add 3894'562 flags = none -24> 2020-03-18 18:39:09.483625 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 flags = none tried to add 5172'1141 flags = none -23> 2020-03-18 18:39:09.483639 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 flags = none tried to add 1915'218 flags = none -22> 2020-03-18 18:39:09.483654 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 flags = delete tried to add 1805'198 flags = delete -21> 2020-03-18 18:39:09.483667 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 flags = delete tried to add 3908'593 flags = delete -20> 2020-03-18 18:39:09.483681 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 flags = none tried to add 5176'1523 flags = none -19> 2020-03-18 18:39:09.483695 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 flags = none tried to add 5165'735 flags = none -18> 2020-03-18 18:39:09.483710 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 flags = none tried to add 3908'563 flags = none -17> 2020-03-18 18:39:09.483724 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 flags = none tried to add 4427'730 flags = none -16> 2020-03-18 18:39:09.483738 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 flags = none tried to add 4743'732 flags = none -15> 2020-03-18 18:39:09.483752 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 flags = none tried to add 5175'1287 flags = none -14> 2020-03-18 18:39:09.483766 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 flags = none tried to add 5172'1144 flags = none -13> 2020-03-18 18:39:09.483782 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 flags = none tried to add 1915'219 flags = none -12> 2020-03-18 18:39:09.483796 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 flags = delete tried to add 3092'344 flags = delete -11> 2020-03-18 18:39:09.483811 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 flags = delete tried to add 3092'381 flags = delete -10> 2020-03-18 18:39:09.483825 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 flags = delete tried to add 1805'197 flags = delete -9> 2020-03-18 18:39:09.483840 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 flags = delete tried to add 1805'199 flags = delete -8> 2020-03-18 18:39:09.483854 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 flags = none tried to add 2197'271 flags = none -7> 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none -6> 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none -5> 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete -4> 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -3> 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -2> 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -1> 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 0> 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** in thread 201e35fdb40 thread_name:tp_osd_tp ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0x145882c) [0x2000245882c] 2: (()+0x19890) [0x2000c54b890] 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] 19: (()+0x80fc) [0x2000c53a0fc] 20: (()+0x119854) [0x2000f2ad854] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. huxiaoyu(a)horebdata.cn

1 0

New Ceph Cluster Setup
by adhobale8＠gmail.com 18 Mar '20

18 Mar '20

Dear Team, We are creating new ceph cluster with three OSD node each node will have 38TB disk space, unfortunately we are having two servers with us third server will get delivered after 3 weeks. I need your help to plan the cluster setup, please suggest which approach will be the right to initiate the setup. 1) Will create the ceph cluster with two osd node by doing "osd pool default size = 2" after that we can add the third node in live cluster and change the replication factor of pool from 2 to 3 by doing # "ceph osd pool set <Pool> size 3". 2) Can we create ceph cluster with 2 osd node by keeping default replication factor 3, and add third osd node in cluster whenever we receive it ?

2 1

bluefs enospc
by Derek Yarnell 18 Mar '20

18 Mar '20

Hi, We have a production cluster that just suffered an issue with multiple of our NVMe OSDs. Multiple of them died (>12) with errors that they no longer had space with a 'ENOSPC from bluestore, misconfigured cluster' error over 4 nodes. These are all simple one device bluestore osds. ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) This is an example[0] of one of the logs. In this case each of 8 NVMe OSDs on a node have 106GB of space allocated to each bluestore NVMe OSD. The ceph-bluestore-tool bluefs-bdev-sizes output only lists 22GiB for osd 681. I extended the space of bluestore on a few of the OSDs via LVM and then the bluefs-bdev-expand command. This worked for a few and not for others. Some of the ones that it did work for recovered for a bit then re-entered the error state. Trying to extend the allocation didn't work after that. When they failed again I ran the fsck which reported that it found 1 error and then running repair I got a rather long stack trace[1]. # ceph-bluestore-tool --log-level 30 --command bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-681 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-681/block -> /dev/dm-33 1 : device size 0x1a80000000 : own 0x[2480000~10000,24a0000~10000,2520000~60000,25f0000~c0000,2720000~50000,28a0000~110000,2a20000~230000,2cc0000~260000,2f30000~220000,31c0000~6b0000,38a0000~10000,3990000~3e0000,3d80000~530000,42d0000~590000,48d0000~400000,4d00000~7d0000,54f0000~c50000,6150000~10000,6190000~150000,6350000~c0000,6480000~160000,6640000~1e0000,6870000~c0000,6a00000~30000,6a40000~240000,6dd0000~310000,7210000~b0000,73a0000~b0000,76a0000~180000,7830000~80000,78e0000~240000,7b70000~90000,7c50000~b0000,7ef0000~140000,8040000~30000,8180000~250000,8440000~50000,84b0000~110000,8610000~c0000,9e20000~20000,9e50000~b0000,9f10000~60000,9f80000~30000,dd80000~180000,df70000~6a0000,e620000~5ae0000,15200000~3510000,187f0000~bf0000,19490000~1070000,1ab70000~4c0000,1b400000~7d0000,1bbe0000~c20000,1cd10000~340000,1d3a0000~860000,1dd00000~2e00000,20c00000~3f00000,24d00000~700000,25600000~700000,26100000~200000,26400000~300000,26b00000~600000,27400000~400000,27ba0000~6e0000,28500000~1d00000,2a400000~700000,2ac00000~100000,2b100000~300000,2b470000~120000,2b700000~500000,2c000000~200000,2c400000~400000,2ca00000~100000,2cf00000~300000,2d340000~39b0000,30d00000~1f00000,32e00000~4bf0000,380a0000~3c0000,38500000~c0000,38bd0000~400000,390b0000~340000,39400000~100000,39900000~1000000,3ac00000~5d00000,40b90000~400000,41280000~db50000,4ee00000~700000,4f900000~4500000,54390000~100000,54e00000~18400000,6d800000~20d0000,6f8f0000~1a10000,71400000~4500000,76100000~300000,766e0000~6860000,7dd00000~c00000,7eac0000~a0000,7ef90000~f190000,8e1f0000~80000,8e410000~60000,8e480000~20000,8e4b0000~20000,8e5c0000~50000,8e7e0000~50000,8f160000~60000,8f240000~a0000,90000000~15e90000,a6200000~c3a0000,b25d0000~630000,b3000000~c00000,b3ee0000~90000,b4200000~d00000,b5a70000~160000,b63f0000~2a0000,b6720000~2820000,bab00000~400000,bbf60000~10ad0000,ccb90000~2300000,cf000000~2b00000,d1ca0000~10000,d1e00000~1400000,d3230000~1df0000,d5200000~1a00000,d6d00000~800000,d75e0000~6f0000,d7f00000~d00000,d9100000~400000,d9900000~d00000,da800000~600000,daf10000~400000,db700000~1600000,dd280000~20000,dd670000~390000,dda30000~400000,de190000~70000,de2a0000~370000,de660000~20000,de700000~14770000,f3600000~700000,f3db0000~960000,f49e0000~5b00000,fa600000~c00000,fb300000~510000,fbb00000~100000,fbeb0000~450000,fc400000~2b0000,fd400000~400000,fde00000~c00000,ff0b0000~50000,ff200000~800000,ffd60000~10000,fff00000~a0000,100200000~300000,101600000~100000,101750000~300000,102120000~1e0000,1027f0000~a00000,103600000~330000,103b00000~200000,103e60000~4a0000,104310000~c00000,105030000~1200000,106800000~100000,106b20000~400000,107000000~300000,1073e0000~400000,107950000~86b0000,110140000~d0000,110350000~2e0000,110e20000~20000,110eb0000~a0000,110f60000~60000,110fd0000~1f0000,1112a0000~f0000,111420000~30000,1115b0000~30000,111620000~150000,111790000~40000,112560000~180000,112730000~180000,1129b0000~50000,112f90000~4a0000,113840000~c0000,113ea0000~40000,113fb0000~130000,114100000~310000,114470000~10000,114620000~120000,114810000~120000,114a00000~20000,114a90000~f0000,114c60000~e0000,114e80000~20000,114f70000~140000,1150c0000~50000,1151f0000~320000,1155f0000~10000,115670000~226f0000,137e30000~800000,138b50000~400000,139400000~1500000,13ae00000~4500000,13f480000~400000,13f950000~1b6e0000,15b700000~400000,15c000000~300000,15c600000~700000,15d9e0000~1820000,15f400000~c00000,160400000~d00000,1613a0000~630000,1619e0000~d20000,162800000~1b00000,164600000~7550000,170d00000~1800000,172580000~100000,172d70000~1c190000,18f100000~40e0000,193700000~400000,193c90000~6970000,19aa90000~188d0000,1b3770000~c220000,1bff00000~1200000,1c12f0000~400000,1c2c00000~400000,1c3f80000~400000,1c5300000~4200000,1c95c0000~b9a40000,283800000~1800000,289000000~3800000,28d000000~6000000,293800000~2000000,296000000~e000000,2a4800000~465d0000,4b6100000~4c560000,502800000~32500000,cb8500000~10f600000,1139800000~7a000000,1668d30000~3d290000,1794000000~3e800000,191c000000~3ee00000,1a49800000~4110000,1a4f800000~40f0000,1a70800000~4100000,1a76000000~3c10000] = 0x582550000 : using 0x56d090000(22 GiB) Any help here would be appreciated, I have stopped out CephFS file system but our radosgw is also impacted. [0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.681.log [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.repair Thanks, derek -- Derek T. Yarnell Director of Computing Facilities University of Maryland Institute for Advanced Computer Studies

2 6

Object storage multisite
by Ignazio Cassano 18 Mar '20

18 Mar '20

Hello, I have two openstack installation on different sites. They do not share any service : each one have its keystone repository, and ceph cluster with object storage and block storage. I read about object storage multisite and I could modify my object storages to enable multisite active-active. I wonder if I can do it with two different keystone . How can I read containers on site B replicated by site A ? Ignazio

1 0

Upmap balancing - pools grouped together?
by Andras Pataki 17 Mar '20

17 Mar '20

I've been trying the upmap balancer on a new Nautilus cluster. We three main pools, a triple replicated pool (id:1) and two 6+3 erasure coded pools (id: 4 and 5). The balancer does a very nice job on the triple replicated pool, but does something strange on the EC pools. Here is a sample of the PG counts on OSDs (each line) per pool (columns): OSD 1 2 3 4 5 ALL 0 25 - - 80 34 139 1 26 - - 74 42 142 2 25 - - 74 42 141 3 26 - - 75 41 142 4 25 - - 83 31 139 5 26 - - 80 35 141 6 26 - - 79 36 141 7 26 - - 72 44 142 8 25 - - 74 42 141 9 26 - - 78 38 142 10 26 - - 70 46 142 11 25 - - 78 36 139 12 26 - - 74 41 141 13 26 - - 78 37 141 14 26 - - 76 40 142 15 25 - - 82 33 140 16 26 - - 77 37 140 17 26 - - 73 43 142 18 26 - - 77 37 140 19 26 - - 79 35 140 20 26 - - 74 42 142 21 26 - - 78 36 140 22 26 - - 79 35 140 23 26 - - 81 34 141 24 26 - - 77 38 141 25 26 - - 79 35 140 Pool #1 has 25 or 26 PGs on each OSD extremely consistently. But it looks like pool #4 and #5 have a lot of variance, but their *sum* seems to be well balanced (and hence the total PGs per OSD is also very consistent). As if the balancer was balancing the two pools together as one. Both pools are EC with 6+3 profile, and use the same crush rule. Could that be the reason? If not - any ideas? Also - a separate balancer related question: is there a way to have the balancer balace the size on each OSD per pool as opposed to the number of PGs? If not currently - is it something hard to implement? Andras

2 1

Inactive PGs
by Peter Eisch 17 Mar '20

17 Mar '20

Full cluster is 14.2.8. I had some OSD drop overnight which results now in 4 inactive PGs. The pools had three participant (2 ssd, 1 sas) OSDs. In each pool at least 1 ssd and 1 sas OSD is working without issue. I’ve ‘ceph pg repair <pg>’ but it doesn’t seem to make any changes. PG_AVAILABILITY Reduced data availability: 4 pgs inactive, 4 pgs incomplete pg 10.2e is incomplete, acting [59,67] pg 10.c3 is incomplete, acting [62,105] pg 10.f3 is incomplete, acting [62,59] pg 10.1d5 is incomplete, acting [87,106] Using `ceph pg <pg> query` I can see the OSD in each case of the ones which failed. Respectively they are: pg 10.2e participants: 59, 68, 77, 143 pg 10.c3 participants: 60, 62, 85, 102, 105, 106 pg 10.f3 participants: 59, 64, 75, 107 pg 10.1d5 participants: 64, 77, 87, 106 The OSDs which are now down/out and have been removed from the crush map and removed the auth are: 62, 64, 68 Of course I have lots of reports of slow OSDs now from OSDs worried about the inactive PGs. How do I properly kick these PGs to have them drop their usage of the OSDs which no longer exist? Thanks for you thoughts on this, peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.64

2 4

High memory ceph mgr 14.2.7
by hoannv46＠gmail.com 17 Mar '20

17 Mar '20

Hi all. My cluster in ceph version 14.2.6 Mgr process in top 3104786 ceph 20 0 20.2g 19.4g 18696 S 315.3 62.0 41:32.74 ceph-mgr All port ceph mgr open netstat -tulpn | grep ceph tcp 0 0 172.25.7.151:6800 0.0.0.0:* LISTEN 3104786/ceph-mgr tcp 0 0 172.25.7.151:6801 0.0.0.0:* LISTEN 3104786/ceph-mgr tcp 0 0 172.25.7.151:3300 0.0.0.0:* LISTEN 3407730/ceph-mon tcp 0 0 172.25.7.151:6789 0.0.0.0:* LISTEN 3407730/ceph-mon All connect to ceph mgr tcp 0 0 172.25.7.154:6800 172.25.5.23:42926 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.5.3:38732 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6801 172.25.5.2:42582 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.5.23:42930 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.6.2:39544 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.5.20:46618 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6801 172.25.5.4:35208 ESTABLISHED 250159/ceph-mgr netstat -plan | grep mgr | wc -l 2422 Connection connect to mgr is osd. How i can reduce mgr memory. Thanks.

6 8

Don't know how to use S3 notification
by jsobczak＠man.poznan.pl 17 Mar '20

17 Mar '20

Hello, I try to set up s3 http notifications but cant seem to figure it out. I tried with python boto3 like this: bucket_notification = s3.BucketNotification('wiaderko0') response = bucket_notification.put( NotificationConfiguration={ 'TopicConfigurations': [ { 'TopicArn': 'Kuba', 'Events': [ 's3:ObjectCreated:*' ] } ] } ) But get the following error: botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unknown I also tried using RGW REST api like with POST like this: http://172.16.97.1:7480/?Action=CreateTopic&Name=test&push-endpoint=http://… But got error like this: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>MethodNotAllowed</Code> <RequestId>tx0000000000000000d1e89-005e707912-11e4-default</RequestId> <HostId>11e4-default-default</HostId> </Error> Could any one offer some advice about it. I would appreciate some guide to setting it up as documentation is pretty unclear.

2 2

Point-in-Time Recovery
by Ml Ml 17 Mar '20

17 Mar '20

Hello List, when reading: https://docs.ceph.com/docs/master/rbd/rbd-mirroring/ it says: (...)Journal-based: This mode uses the RBD journaling image feature to ensure point-in-time, crash-consistent replication between clusters(...) Does this mean, that mean, that we have some kind of transaction logs where we will be able to "walk to" a specific time to restore a specific state? Like: https://blog.sleeplessbeastie.eu/2016/02/29/how-to-perform-postgresql-point… Thanks, Michael

2 1

For urgent help: OSD down under heavier workload
by huxiaoyu＠horebdata.cn 17 Mar '20

17 Mar '20

Dear Ceph folks, Can someoen with better ceph expertise help out? I am running a small ceph cluster of 3 nodes, Luminoous 12.2.12. On one node with weaker CPU, i have to reweight OSDs to keep them alive. If PGs on these OSDs exceeds some 40+, then these OSDs will continusouly restart (down) without success. I checked the CPU usage and it is actually not overloaded. What could be the root problem? best regards, Samuel *********************************************************************************************************************************************** lags = delete -42> 2020-03-17 17:13:53.374486 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf69ccd:::rbd_data.128dd6b8b4567.0000000000003e96:head have 217'1230 flags = delete tried to add 217'1230 flags = delete -41> 2020-03-17 17:13:53.374502 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf72b1e:::rbd_data.128f46b8b4567.00000000000005e4:head have 217'1103 flags = delete tried to add 217'1103 flags = delete -40> 2020-03-17 17:13:53.374518 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf91ecf:::rbd_data.8f3b66b8b4567.000000000000038e:head have 3093'11873 flags = delete tried to add 3093'11873 flags = delete -39> 2020-03-17 17:13:53.374533 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf9b36e:::rbd_data.2a0286b8b4567.0000000000000176:head have 1619'2253 flags = delete tried to add 1619'2253 flags = delete -38> 2020-03-17 17:13:53.374549 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf9dcdd:::rbd_data.128f46b8b4567.00000000000019b8:head have 217'1145 flags = delete tried to add 217'1145 flags = delete -37> 2020-03-17 17:13:53.374565 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfa7411:::rbd_data.73a8f6b8b4567.0000000000000231:head have 1934'8269 flags = delete tried to add 1934'8269 flags = delete -36> 2020-03-17 17:13:53.374581 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfc02e9:::rbd_data.128dd6b8b4567.00000000000002d0:head have 217'1111 flags = delete tried to add 217'1111 flags = delete -35> 2020-03-17 17:13:53.374597 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfccec4:::rbd_data.128f46b8b4567.0000000000000ea6:head have 217'1125 flags = delete tried to add 217'1125 flags = delete -34> 2020-03-17 17:13:53.374612 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfcf45a:::rbd_data.74a826b8b4567.000000000000004a:head have 1934'10159 flags = none tried to add 1934'10159 flags = none -33> 2020-03-17 17:13:53.374628 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfeb6f9:::rbd_data.8d7246b8b4567.00000000000042a5:head have 4114'12725 flags = none tried to add 4114'12725 flags = none -32> 2020-03-17 17:13:53.374643 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfeb9ce:::rbd_data.6e6fa6b8b4567.0000000000000093:head have 1934'10160 flags = none tried to add 1934'10160 flags = none -31> 2020-03-17 17:13:53.374659 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbff1dd0:::rbd_data.128dd6b8b4567.0000000000004332:head have 217'1240 flags = delete tried to add 217'1240 flags = delete -30> 2020-03-17 17:13:53.374675 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbffd7b5:::rbd_data.128dd6b8b4567.0000000000004330:head have 217'1239 flags = delete tried to add 217'1239 flags = delete -29> 2020-03-17 17:13:54.738477 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c0b4c0:::rbd_data.9177446e87ccd.0000000000000b1d:head have 4711'802 flags = none tried to add 4711'802 flags = none -28> 2020-03-17 17:13:54.738516 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c185d7:::rbd_data.589572ae8944a.0000000000000c3a:head have 1805'339 flags = delete tried to add 1805'339 flags = delete -27> 2020-03-17 17:13:54.738535 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c1b3ab:::rbd_data.8922074b0dc51.0000000000003ea1:head have 3099'603 flags = delete tried to add 3099'603 flags = delete -26> 2020-03-17 17:13:54.738552 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c28113:::rbd_data.9117a327b23c6.00000000000002bd:head have 3122'646 flags = none tried to add 3122'646 flags = none -25> 2020-03-17 17:13:54.738568 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c2926a:::rbd_data.589572ae8944a.000000000000022d:head have 1805'337 flags = delete tried to add 1805'337 flags = delete -24> 2020-03-17 17:13:54.738584 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c2bb8b:::rbd_data.5f15f625558ec.000000000000027f:head have 1913'383 flags = none tried to add 1913'383 flags = none -23> 2020-03-17 17:13:54.738600 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c3ca0e:::rbd_data.8a738625558ec.0000000000005ca0:head have 3298'654 flags = none tried to add 3298'654 flags = none -22> 2020-03-17 17:13:54.738615 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c4892e:::rbd_data.8ff1766334873.0000000000009ea0:head have 3092'523 flags = delete tried to add 3092'523 flags = delete -21> 2020-03-17 17:13:54.738631 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c4ce18:::rbd_data.9177446e87ccd.0000000000000d58:head have 4750'803 flags = none tried to add 4750'803 flags = none -20> 2020-03-17 17:13:54.738647 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c675ec:::rbd_data.8ff1766334873.00000000000018a2:head have 3092'468 flags = delete tried to add 3092'468 flags = delete -19> 2020-03-17 17:13:54.738662 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18cbb5d5:::rbd_data.9117a327b23c6.00000000000028a6:head have 3126'652 flags = none tried to add 3126'652 flags = none -18> 2020-03-17 17:13:54.738678 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d2a04f:::rbd_data.110706b8b4567.0000000000000662:head have 1601'112 flags = none tried to add 1601'112 flags = none -17> 2020-03-17 17:13:54.738693 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d424df:::rbd_data.9177446e87ccd.0000000000001ca7:head have 3122'649 flags = none tried to add 3122'649 flags = none -16> 2020-03-17 17:13:54.738708 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d43cd1:::rbd_data.589572ae8944a.0000000000003a12:head have 1805'350 flags = delete tried to add 1805'350 flags = delete -15> 2020-03-17 17:13:54.738724 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d91ef4:::rbd_data.589572ae8944a.0000000000004d34:head have 1805'354 flags = delete tried to add 1805'354 flags = delete -14> 2020-03-17 17:13:54.738739 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18e2953a:::rbd_data.5f15f625558ec.00000000000089d0:head have 1913'384 flags = none tried to add 1913'384 flags = none -13> 2020-03-17 17:13:54.738755 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18e514a7:::rbd_data.8f17c327b23c6.00000000000040a3:head have 2962'410 flags = delete tried to add 2962'410 flags = delete -12> 2020-03-17 17:13:54.738770 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18ea2e7b:::rbd_data.9177446e87ccd.000000000000095c:head have 4711'801 flags = none tried to add 4711'801 flags = none -11> 2020-03-17 17:13:54.738785 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18ed4244:::rbd_data.8ff1766334873.00000000000006a2:head have 3092'456 flags = delete tried to add 3092'456 flags = delete -10> 2020-03-17 17:13:54.738801 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f0ccd9:::rbd_data.589572ae8944a.00000000000045e7:head have 1805'353 flags = delete tried to add 1805'353 flags = delete -9> 2020-03-17 17:13:54.738816 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f69445:::rbd_data.589572ae8944a.0000000000000229:head have 1805'336 flags = delete tried to add 1805'336 flags = delete -8> 2020-03-17 17:13:54.738833 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f88400:::rbd_data.589572ae8944a.0000000000004e31:head have 1805'355 flags = delete tried to add 1805'355 flags = delete -7> 2020-03-17 17:13:54.738848 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f8e666:::rbd_data.9177446e87ccd.0000000000000387:head have 4416'800 flags = none tried to add 4416'800 flags = none -6> 2020-03-17 17:13:54.738864 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18fae8ee:::rbd_data.9177446e87ccd.0000000000005aa6:head have 3330'657 flags = none tried to add 3330'657 flags = none -5> 2020-03-17 17:13:54.738880 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18fdcb08:::rbd_data.110706b8b4567.0000000000001401:head have 1601'113 flags = none tried to add 1601'113 flags = none -4> 2020-03-17 17:14:19.091232 2000f81db40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.11:0/1871093011 conn(0x20014cce840 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -3> 2020-03-17 17:14:19.106328 2000ec51b40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.12:0/4148513923 conn(0x20014109640 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -2> 2020-03-17 17:14:19.155177 2000e3ddb40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.13:0/768399090 conn(0x200141ac4d0 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -1> 2020-03-17 17:14:36.138777 2000f81db40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.202:0/3091162658 conn(0x200147d1f90 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 0> 2020-03-17 17:15:06.927121 201a6afdb40 -1 *** Caught signal (Bus error) ** in thread 201a6afdb40 thread_name:tp_osd_tp ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0x145882c) [0x2000245882c] 2: (()+0x19890) [0x2000d281890] 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] 8: (ReplicatedBackend::_do_pull_response(boost::intrusive_ptr<OpRequest>)+0x650) [0x2000203e5f0] 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x3a4) [0x200020440c4] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] 19: (()+0x80fc) [0x2000d2700fc] 20: (()+0x119854) [0x2000bed1854] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. huxiaoyu(a)horebdata.cn

1 0

OSD failing to restart with "no available blob id"
by Gilles Mocellin 16 Mar '20

16 Mar '20

Hi ! I'm stuck with "no available blob id" during the start of an OSD. It seems there's a workaround back-ported only in nautilus (Bug https://tracker.ceph.com/issues/38272) but I use mimic for now. Someone has an operational workaround ? Or should I recreate my OSD ? And what is the easiest way to do that ? Thanks, --- Gilles M

1 0

Forcibly move PGs from full to empty OSD
by Thomas Schneider 16 Mar '20

16 Mar '20

Hi, Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too. Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)". Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? THX

5 8

upmap balancer
by Frank R 16 Mar '20

16 Mar '20

Hi all, Does using the Upmap balancer require that all OSDs be the same size (per device class)? thx Frank

4 7

Advice on sizing WAL/DB cluster for Optane and SATA SSD disks.
by Victor Hooi 16 Mar '20

16 Mar '20

Hi, I'm building a 4-node Proxmox cluster, with Ceph for the VM disk storage. On each node, I have: - 1 x 512Gb M.2 SSD (for Proxmox/boot volume) - 1 x 960GB Intel Optane 905P (for Ceph WAL/DB) - 6 x 1.92TB Intel S4610 SATA SSD (for Ceph OSD) I'm using the Proxmox "pveceph" command to setup the OSDs. By default this seems to pick 10% of the OSD size for the DB volume, and 1% of the OSD size for the WAL volume. This means after four drives, I ran out of space: # pveceph osd create /dev/sde -db_dev /dev/nvme0n1 > create OSD on /dev/sde (bluestore) > creating block.db on '/dev/nvme0n1' > Rounding up size to full physical extent 178.85 GiB > lvcreate > 'ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee/osd-db-da591d0f-8a05-42fa-bc62-a093bf98aded' > error: Volume group "ceph-861ebf6d-8fee-4313-8de6-4e797dc436ee" has > insufficient free space (45784 extents): 45786 required. Anyway, I assume that means I need to tune my DB and WAL volumes down from the defaults. What advice to you have in terms of making best use of the available space, between WAL and DB? What is the impact of having WAL and DB smaller than 1% and 10% of OSD size respectively? Thanks, Victor

5 5

HEALTH_WARN 1 pools have too few placement groups
by Dietmar Rieder 16 Mar '20

16 Mar '20

Hi, I was planing to activate the pg_autoscaler on a EC (6+3) pool which I created two years ago. Back then I calculated the total # of pgs for this pool with a target per ods pg # of 150 (this was the recommended /osd pg number as far as I recall). I used the RedHat ceph pg per pool calculator [1] with the following settings: PoolyType: EC K: 6 M: 3 OSD #: 220 %Data: 100.00 Target PGs per OSD: 150 This resulted into a suggested PG count of 4096, which I used to create that pool. (BTW: I get the same using the ceph.io pgcalc [2], setting size to 9 (=EC 6+3)) # ceph osd df shows that I have now 162-169 pgs / osd Now I enabled the pg_autoscaler and set autoscale_mode to "warn", using the following commands: # ceph mgr module enable pg_autoscaler # ceph osd pool set hdd-ec-data-pool pg_autoscale_mode warn when i set the target_ratio_size to 1, using: # ceph osd pool set hdd-ec-data-pool target_size_ratio 1 I get the warning: 1 pools have too few placement groups The autoscale status shows that pg number in the EC pool would get scaled up to 16384 pgs: # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE ssd-rep-metadata-pool 11146M 3.0 29805G 0.0011 1.0 64 4 off ssd-rep-data-pool 837.8G 3.0 29805G 0.0843 1.0 1024 64 off hdd-ec-data-pool 391.3T 1.5 1614T 0.3636 1.0000 1.0 4096 16384 warn How does this relate the the calculated number of 4096 pgs for that EC pool? I mean 16384 is quite a jump from the original value. Is the autoscaler miscalculating something or is the pg calculator wrong? Can someone explain it? (My cluster is on ceph version 14.2.7) Thanks so much Dietmar [1]: https://access.redhat.com/labs/cephpgc/ [2]: https://ceph.io/pgcalc/ -- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Email: dietmar.rieder(a)i-med.ac.at Web: http://www.icbi.at

2 2

Zabbix module failed to send data - SSL support
by tdados＠hotmail.com 16 Mar '20

16 Mar '20

Hello all, We are having an issue with the Ceph Zabbix module and it's failing to send data. The reason is that in our Zabbix infrastructure we use Encryption and agent connection with certificate as well. I see the logs that are failing due to that reason from the zabbix proxy servers. 1329:20200312:092834.705 connection of type "unencrypted" is not allowed for host "xxx" - this is ceph host 1328:20200312:092840.337 connection of type "unencrypted" is not allowed for host "xxx" 1329:20200312:092909.423 connection of type "unencrypted" is not allowed for host "xxx" In the zabbix ceph config from ceph there is no configuration option to allow ssl. I also tried different options with the sender like to use the configuration file like ""ceph zabbix config-set zabbix_sender "/usr/bin/zabbix_sender -c /etc/zabbix/zabbix_agentd.conf"" but didn't work either. Anyone has any tips how to tell from ceph config to allow/enable ssl on the zabbix module?

1 0

Trying to follow installation documentation
by Mark M 16 Mar '20

16 Mar '20

Hi List, Trying to learn Ceph, so I'm using a Vagrant setup of six nodes with Docker and LVM on them. But as soon as I reach this step https://docs.ceph.com/docs/master/cephadm/#bootstrap-a-new-cluster I get stuck with this error: ``` INFO:cephadm:Mgr epoch 5 not available, waiting (6/10)... INFO:cephadm:Non-zero exit code 22 from /bin/docker run --rm --net=host -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor01 -v /var/log/ceph/114dc730-66ca -11ea-b6f9-5254008afee6:/var/log/ceph:z -v /tmp/ceph-tmpbrsg0raa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq7q4mc5h:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph /daemon-base:latest-master-devel tell mgr mgr_status INFO:cephadm:/usr/bin/ceph:stderr no valid command found; 10 closest matches: INFO:cephadm:/usr/bin/ceph:stderr 0 INFO:cephadm:/usr/bin/ceph:stderr 1 INFO:cephadm:/usr/bin/ceph:stderr 2 INFO:cephadm:/usr/bin/ceph:stderr abort INFO:cephadm:/usr/bin/ceph:stderr assert INFO:cephadm:/usr/bin/ceph:stderr config diff INFO:cephadm:/usr/bin/ceph:stderr config diff get <var> INFO:cephadm:/usr/bin/ceph:stderr config get <var> INFO:cephadm:/usr/bin/ceph:stderr config help [<var>] INFO:cephadm:/usr/bin/ceph:stderr config set <var> <val>... INFO:cephadm:/usr/bin/ceph:stderr Error EINVAL: invalid command INFO:cephadm:Mgr epoch 5 not available, waiting (7/10)... ``` And then it just repeats. Also the "/var/log/ceph/114dc730-66ca-11ea-b6f9-5254008afee6" folder is empty, no logs. I must admit I'm stumped, I don't really know where to go from here. Any pointers for me perhaps? Thanks! Mark

1 1

Re: Kworker 100% with ceph-msgr (after upgrade to 14.2.6?)
by Ilya Dryomov 15 Mar '20

15 Mar '20

On Tue, Jan 14, 2020 at 10:31 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I think this is new since I upgraded to 14.2.6. kworker/7:3 100% > > [@~]# echo l > /proc/sysrq-trigger > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192e7e>] > [<ffffffffb0192e7e>] generic_swap+0x1e/0x30 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a20 EFLAGS: 00000206 > [Tue Jan 14 10:05:08 2020] RAX: 0000000000000072 RBX: 0000000000000060 > RCX: 0000000000000072 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000006 RSI: ffffa0c788914a7a > RDI: ffffa0c788914a42 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a20 R08: 0000000000000028 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe94984224500 > R12: ffffa0c788914a40 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0c788914a18 > R15: ffffa0c788914a78 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > > > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192200>] > [<ffffffffb0192200>] __x86_indirect_thunk_rax+0x0/0x20 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a28 EFLAGS: 00000286 > [Tue Jan 14 10:05:08 2020] RAX: ffffffffb0192e60 RBX: 0000000000000010 > RCX: 0000000000000010 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000008 RSI: ffffa0d0e108b828 > RDI: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a98 R08: 0000000000000000 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe949a9842280 > R12: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0d0e108b818 > R15: ffffa0d0e108b828 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] ? sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 Hi Marc, It's busy processing snapshot contexts, I think it's more or less the same issue that you reported in "ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)". Did "ceph: build_snap_context 100020c9287 ffff911a9a26bd00 fail -12" errors go away? How many snapshots do you have now? Did you file a tracker ticket? It's better to have all details in one place rather than scattered across different threads on the list. Adding Jeff and Zheng, who can take a closer look. Thanks, Ilya

2 2

Weird monitor and mgr behavior after update.
by Cassiano Pilipavicius 15 Mar '20

15 Mar '20

Hello, I have updated my cluster from luminous to nautilus and now my cluster is working but I am seeing a weird behavior in my monitors and managers. The monitors are using a huge amount of memory and becoming very slow. The CPU usage is also very higher than it used to be. The manager keeps constantly restarting and in many opportunities it report all zero values in the pg numbers, space usage etc. I also started receiving many slow ops warning from monitors. Does any one has seen something like this? Which info do I need to send about my cluster so I can get some help from more experienced users? This is one fo the weird log messages: 2020-03-15 06:26:39.474 7f0396cae700 -1 mon.cwvh2@0(leader) e27 get_health_metrics reporting 603796 slow ops, oldest is pool_op(create unmanaged snap pool 5 tid 1236847 name v382060) cluster: id: 4ed305d4-5847-4e63-b9bb-168361cf2e81 health: HEALTH_WARN 1/3 mons down, quorum cwvh8,cwvh15 services: mon: 3 daemons, quorum (age 4M), out of quorum: cwvh2, cwvh8, cwvh15 mgr: cwvh13(active, since 33s), standbys: cwvh14 osd: 100 osds: 100 up (since 17h), 100 in data: pools: 6 pools, 4160 pgs objects: 19.35M objects, 64 TiB usage: 132 TiB used, 94 TiB / 226 TiB avail pgs: 4154 active+clean 6 active+clean+scrubbing+deep io: client: 192 MiB/s rd, 52 MiB/s wr, 1.25k op/s rd, 1.34k op/s wr [root@cwvh15 ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED backup 120 TiB 62 TiB 58 TiB 58 TiB 48.64 hdd 12 TiB 4.3 TiB 7.8 TiB 7.8 TiB 64.47 ssd 94 TiB 28 TiB 66 TiB 66 TiB 70.40 TOTAL 226 TiB 94 TiB 132 TiB 132 TiB 58.51 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL rbd 0 19 B 2 128 KiB 0 11 TiB CWVHDS 1 420 GiB 110.03k 848 GiB 39.36 653 GiB COMP 4 2.7 TiB 717.72k 5.5 TiB 81.19 653 GiB SSD 5 35 TiB 9.43M 66 TiB 88.08 4.4 TiB HDD 6 730 GiB 196.24k 1.4 TiB 52.84 653 GiB BKPR1 8 33 TiB 8.90M 58 TiB 56.73 22 TiB

1 0

How to get num ops blocked per OSD
by Robert LeBlanc 14 Mar '20

14 Mar '20

For Jewel I wrote a script to take the output of `ceph health detail --format=json` and send alerts to our system that ordered the osds based on how long the ops were blocked and which OSDs had the most ops blocked. This was really helpful to quickly identify which OSD out of a list of 100 would be the most probable one having issues. Since upgrading to Luminous, I don't get that and I'm not sure where that info went to. Do I need to query the manager now? This is the regex I was using to extract the pertinent information: '^(\d+) ops are blocked > (\d+\.+\d+) sec on osd\.(\d+)$' Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 2

rgw.none shows extremely large object count
by yuezhu3＠gmail.com 14 Mar '20

14 Mar '20

Hello, we are running a ceph cluster + rgw on luminous 12.2.12 that serves as a S3 compatible storage. We have noticed some buckets where the `rgw.none` from the output of `radosgw-admin bucket stats` shows extremely large value for `num_objects`, which is not convincible. It does look like an underflow by subtracting a positive number from 0 and then the value is interpreted and shown as an uint64. For example, ``` # radosgw-admin bucket stats --bucket redacted { "bucket": "redacted", ........... "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551607 }, "rgw.main": { "size": 1687971465874, "size_actual": 1696692400128, "size_utilized": 1687971465874, "size_kb": 1648409635, "size_kb_actual": 1656926172, "size_kb_utilized": 1648409635, "num_objects": 4290147 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 75 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } ``` We did find a few reports on this issue, eg. http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-November/037531.ht…. Do we know any use patterns that can lead object count to become that large? Also is there a way to accurately collect the object count for each bucket in the cluster, as we would like to use it for management purpose.

1 0

Re: Is there a better way to make a samba/nfs gateway? (Marc Roos)
by Chad William Seys 14 Mar '20

14 Mar '20

Awhile back I thought there were some limitations which prevented us from trying this, but I cannot remember... What does the ceph vfs gain you over exporting by cephfs kernel module (kernel 4.19). What does it lose you? (I.e. pros and cons versus kernel module?) Thanks! C. > It's based on vfs_ceph and you can read more about how to configure it > yourself on > https://www.samba.org/samba/docs/current/man-html/vfs_ceph.8.html.

2 1

Re: Is there a better way to make a samba/nfs gateway?
by Konstantin Shalygin 14 Mar '20

14 Mar '20

On 3/13/20 10:47 PM, Chip Cox wrote: > Konstantin - in your Windows environment, would it be beneficial to > have the ability to have NTFS data land as S3 (object store) on a Ceph > storage appliance? Or does it have to be NFS? > > Thanks and look forward to hearing back. Nope, for windows we use CephFS over Samba VFS CTDB cluster. k

1 0

Possible bug with rbd export/import?
by Matt Dunavant 13 Mar '20

13 Mar '20

Hello, I think I've been running into an rbd export/import bug and wanted to see if anybody else had any experience. We're using rbd images for VM drives both with and without custom stripe sizes. When we try to export/import the drive to another ceph cluster, the VM always comes up in a busted state it can't recover from. This happens both when doing this export/import through stdin/stdout and when using a middle machine as a temp space. I remember doing this a few times in previous versions without error, so I'm not sure if this is a regression or I'm doing something different. I'm still testing this to try and track down where the issue is but wanted to post this here to see if anybody else has any experience. Example command: rbd -c /etc/ceph/cluster1.conf export pool/testvm.boot - | rbd -c /etc/ceph/cluster2.conf import - pool/testvm.boot Current cluster is on 14.2.8 and using Ubuntu 18.04 w/ 5.3.0-40-generic. Let me know if I can provide any more details to help track this down. Thanks,

4 11

MGRs failing once per day and generally slow response times
by Janek Bevendorff 13 Mar '20

13 Mar '20

Hi, I've always had some MGR stability issues with daemons crashing at random times, but since the upgrade to 14.2.8 they regularly stop responding after some time until I restart them (which I have to do at least once a day). I noticed right after the upgrade that the prometheus module was entirely unresponsive and ceph fs status took about half a minute to return. Once all the cluster chatter had settled and the PGs had been rebalanced (auto-scale was messing with PGs after the upgarde), it became usable again, but everything's still slower than before. Prometheus takes several seconds to list metrics, ceph fs status takes about 1-2 seconds. However, after some time, MGRs stop responding and are kicked from the list of standbys. With log level 5 all they are writing to the log files is this: 2020-03-11 09:30:40.539 7f8f88984700 4 mgr[prometheus] ::ffff:xxx.xxx.xxx.xxx - - [11/Mar/2020:09:30:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.15.2" 2020-03-11 09:30:41.371 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:43.392 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:45.412 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:47.436 7f8f9ee62700 4 mgr send_beacon standby 2020-03-11 09:30:49.460 7f8f9ee62700 4 mgr send_beacon standby I have seen another email on this list complaining about slow ceph fs status, I believe this issue is connected. Besides the standard always-on modules I have enabled the prometheus, dashboard, and telemetry modules. Best Janek

4 6

Cancelled: Ceph Day Oslo May 13th
by Wido den Hollander 13 Mar '20

13 Mar '20

Hi, Due to the recent developments around the COVID-19 virus we (the organizers) have decided to cancel the Ceph Day in Oslo on May 13th. Altough it's still 8 weeks away we don't know how the situation will develop and if travel will be possible or people are willing to travel. Therefor we thought it was best to cancel the event for now and to re-schedule to a later date in 2020. We haven't picked a date yet. Once chosen we'll communicate it through the regular channels. Wido

1 0

Re: Ceph Performance of Micron 5210 SATA?
by Marc Roos 13 Mar '20

13 Mar '20

If you are asking, maybe run this? [global] ioengine=libaio invalidate=1 ramp_time=30 iodepth=1 runtime=180 time_based direct=1 filename=/dev/sdf [write-4k-seq] stonewall bs=4k rw=write #write_bw_log=sdx-4k-write-seq.results #write_iops_log=sdx-4k-write-seq.results [randwrite-4k-seq] stonewall bs=4k rw=randwrite #write_bw_log=sdx-4k-randwrite-seq.results #write_iops_log=sdx-4k-randwrite-seq.results [read-4k-seq] stonewall bs=4k rw=read #write_bw_log=sdx-4k-read-seq.results #write_iops_log=sdx-4k-read-seq.results [randread-4k-seq] stonewall bs=4k rw=randread #write_bw_log=sdx-4k-randread-seq.results #write_iops_log=sdx-4k-randread-seq.results [rw-4k-seq] stonewall bs=4k rw=rw #write_bw_log=sdx-4k-rw-seq.results #write_iops_log=sdx-4k-rw-seq.results [randrw-4k-seq] stonewall bs=4k rw=randrw #write_bw_log=sdx-4k-randrw-seq.results #write_iops_log=sdx-4k-randrw-seq.results [write-128k-seq] stonewall bs=128k rw=write #write_bw_log=sdx-128k-write-seq.results #write_iops_log=sdx-128k-write-seq.results [randwrite-128k-seq] stonewall bs=128k rw=randwrite #write_bw_log=sdx-128k-randwrite-seq.results #write_iops_log=sdx-128k-randwrite-seq.results [read-128k-seq] stonewall bs=128k rw=read #write_bw_log=sdx-128k-read-seq.results #write_iops_log=sdx-128k-read-seq.results [randread-128k-seq] stonewall bs=128k rw=randread #write_bw_log=sdx-128k-randread-seq.results #write_iops_log=sdx-128k-randread-seq.results [rw-128k-seq] stonewall bs=128k rw=rw #write_bw_log=sdx-128k-rw-seq.results #write_iops_log=sdx-128k-rw-seq.results [randrw-128k-seq] stonewall bs=128k rw=randrw #write_bw_log=sdx-128k-randrw-seq.results #write_iops_log=sdx-128k-randrw-seq.results [write-1024k-seq] stonewall bs=1024k rw=write #write_bw_log=sdx-1024k-write-seq.results #write_iops_log=sdx-1024k-write-seq.results [randwrite-1024k-seq] stonewall bs=1024k rw=randwrite #write_bw_log=sdx-1024k-randwrite-seq.results #write_iops_log=sdx-1024k-randwrite-seq.results [read-1024k-seq] stonewall bs=1024k rw=read #write_bw_log=sdx-1024k-read-seq.results #write_iops_log=sdx-1024k-read-seq.results [randread-1024k-seq] stonewall bs=1024k rw=randread #write_bw_log=sdx-1024k-randread-seq.results #write_iops_log=sdx-1024k-randread-seq.results [rw-1024k-seq] stonewall bs=1024k rw=rw #write_bw_log=sdx-1024k-rw-seq.results #write_iops_log=sdx-1024k-rw-seq.results [randrw-1024k-seq] stonewall bs=1024k rw=randrw #write_bw_log=sdx-1024k-randrw-seq.results #write_iops_log=sdx-1024k-randrw-seq.results -----Original Message----- From: mj [mailto:lists@merit.unu.edu] Sent: 06 March 2020 10:01 To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Ceph Performance of Micron 5210 SATA? Last monday I performed a quick test with those two disks already, probably not that relevant, but posting it anyway: I created a two-disk ceph 'cluster' on just the one local node, and ran the following: > root@ceph:~# rados bench -p scbench 10 write --no-cleanup hints = 1 > Maintaining 16 concurrent writes of 4194304 bytes to objects of size > 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ceph_48906 > sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) > 0 0 0 0 0 0 - 0 > 1 16 107 91 363.961 364 0.0546517 0.155121 > 2 16 206 190 379.948 396 0.1529 0.159227 > 3 16 324 308 410.614 472 0.0972163 0.151421 > 4 16 458 442 441.942 536 0.0484349 0.141799 > 5 16 590 574 459.141 528 0.0445051 0.136922 > 6 16 727 711 473.941 548 0.181066 0.134468 > 7 16 856 840 479.941 516 0.187683 0.133199 > 8 16 970 954 476.942 456 0.070753 0.132642 > 9 16 1089 1073 476.831 476 0.193608 0.133754 > 10 16 1214 1198 479.142 500 0.0999212 0.132529 > Total time run: 10.097218 > Total writes made: 1215 > Write size: 4194304 > Object size: 4194304 > Bandwidth (MB/sec): 481.321 > Stddev Bandwidth: 60.481 > Max bandwidth (MB/sec): 548 > Min bandwidth (MB/sec): 364 > Average IOPS: 120 > Stddev IOPS: 15 > Max IOPS: 137 > Min IOPS: 91 > Average Latency(s): 0.132889 > Stddev Latency(s): 0.0645579 > Max latency(s): 0.336118 > Min latency(s): 0.0117049 Do let me know what else you'd want me to do. MJ _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 4

ceph qos
by 展荣臻（信泰） 13 Mar '20

13 Mar '20

Hi everyone: There are two qos in ceph(one based on tokenbucket algorithm,another based on mclock ). Which one I can use in production environment？ Thank you

1 0

EC pool 4+2 - failed to guarantee a failure domain
by Maks Kowalik 13 Mar '20

13 Mar '20

Hello, I have created a small 16pg EC pool with k=4, m=2. Then I applied following crush rule to it: rule test_ec { id 99 type erasure min_size 5 max_size 6 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step choose indep 3 type host step chooseleaf indep 2 type osd step emit } The OSD tree looks as following: -1 43.38448 root default -9 43.38448 region lab1 -7 43.38448 room dc1.lab1 -5 43.38448 rack r1.dc1.lab1 -3 14.44896 host host1.r1.dc1.lab1 6 hdd 3.63689 osd.6 up 1.00000 1.00000 8 hdd 3.63689 osd.8 up 1.00000 1.00000 7 hdd 3.63689 osd.7 up 1.00000 1.00000 11 hdd 3.53830 osd.11 up 1.00000 1.00000 -11 14.44896 host host2.r1.dc1.lab1 4 hdd 3.63689 osd.4 up 1.00000 1.00000 9 hdd 3.63689 osd.9 up 1.00000 1.00000 5 hdd 3.63689 osd.5 up 1.00000 1.00000 10 hdd 3.53830 osd.10 up 1.00000 1.00000 -13 14.48656 host host3.r1.dc1.lab1 0 hdd 3.57590 osd.0 up 1.00000 1.00000 1 hdd 3.63689 osd.1 up 1.00000 1.00000 2 hdd 3.63689 osd.2 up 1.00000 1.00000 3 hdd 3.63689 osd.3 up 1.00000 1.00000 My expectation was that each host will contain 2 shards of any PG of the pool. When I dumped PGs, it was true, but one group is placed on OSDs 0,2,3 which will cause downtime in case of host3 failure. root@host1:~/mkw # ceph pg dump|grep "^66\."|awk '{print $17}' dumped all [4,5,7,6,1,2] [8,11,9,3,0,2] <<< - this one is problematic [6,7,10,9,2,0] [2,3,7,6,5,9] [7,8,10,5,3,1] [4,5,8,6,0,2] [7,11,9,4,1,2] [5,9,0,2,7,11] [9,5,3,1,7,8] [8,11,2,0,5,9] [2,0,8,6,10,9] [3,2,5,9,7,11] [6,7,9,5,1,2] [10,5,1,3,11,8] [4,5,7,8,2,0] [7,8,3,2,9,10] Is there a way to ensure that host failure is not disruptive to the cluster? During the experiment I used info from this thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030227.html Kind regards, Maks Kowalik

2 1

Ceph storage distribution between pools
by alexander.v.litvak＠gmail.com 13 Mar '20

13 Mar '20

I have a small cluster with a single crash map. I use 3 pools one (Openebula VMs on rbd), cephfs_data and cephfs_metadata for cephfs. Here is my ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 94 TiB 78 TiB 17 TiB 17 TiB 17.75 TOTAL 94 TiB 78 TiB 17 TiB 17 TiB 17.75 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 3.3 TiB 6.62M 10 TiB 12.36 24 TiB cephfs_metadata 2 2.1 GiB 447.63k 2.5 GiB 0 24 TiB one 5 2.2 TiB 598.12k 6.6 TiB 8.42 24 TiB What confuses me is an even distribution of MAX_AVAIL storage between those pools. When I mount cephfs on a client host, df -h shows me pool utilization. 28T 3.4T 24T 13% I also have an old hammer cluster where I see a similar picture for ceph df for a one crash map (covering rbd, cephfs-data, cephfs-metadata) GLOBAL: SIZE AVAIL RAW USED %RAW USED 87053G 31306G 55747G 64.04 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 12907G 69.37 5700G 3312474 cephfs-data 12 2873G 33.52 5700G 5859947 cephfs-meta 13 90035k 0 5700G 443961 cloud12g 14 2857G 43.41 3726G 623737 However, df -h on clients show total cluster utilization 86T 55T 31T 65% It seems that the hammer dynamically changes allocation data between the pools on the same crush map as needed. Does nautilus do the same? In this case, does 24 TB means actually avaialble space divided 3 (all my pools are set with 3/2 replication)? Thank you and sorry for the confusion

1 0

preventing the spreading of corona virus on ceph.io
by Marc Roos 12 Mar '20

12 Mar '20

Since the spreading of the corona virus is taking such drastic proportions that flights between Europe and the US are being halted. I would suggest we show some support and we temporary use in the mailing list only :) and not :D

1 0

centos7 / nautilus where to get kernel 5.5 from?
by Marc Roos 12 Mar '20

12 Mar '20

I have default centos7 setup with nautilus. I have been asked to install 5.5 to check a 'bug'. Where should I get this from? I read that the elrepo kernel is not compiled like rhel.

2 2

Re: Single machine / multiple monitors
by Brian Topping 12 Mar '20

12 Mar '20

Ok, I think that answers my question then, thanks! Too risky to be playing with patterns that will get increasingly difficult to support over time. > On Mar 12, 2020, at 12:48 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > They won’t be AFAIK. Few people ever did this. > >> On Mar 12, 2020, at 11:08 AM, Brian Topping <brian.topping(a)gmail.com> wrote: >> >> If the ceph roadmap is getting rid of named clusters, how will multiple clusters be supported? How (for instance) would `/var/lib/ceph/mon/{name}` directories be resolved?

1 0

Re: Single machine / multiple monitors
by Brian Topping 12 Mar '20

12 Mar '20

If the ceph roadmap is getting rid of named clusters, how will multiple clusters be supported? How (for instance) would `/var/lib/ceph/mon/{name}` directories be resolved? > On Mar 11, 2020, at 8:29 PM, Brian Topping <brian.topping(a)gmail.com> wrote: > >> On Mar 11, 2020, at 7:59 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: >> >>> This is all possible with a single cluster, but this limited node also needs storage. >> >> Are you saying that the limited node needs to access Ceph-based storage? Is this some sort of converged architecture? > > It is a converged architecture in that all three boxes are running Kubernetes. There is one k8s cluster on each side of the link, let’s call them “primary” and “secondary”: > * The primary k8s cluster will only access storage from the primary Ceph cluster, secondary k8s only accesses storage from secondary Ceph. > * Primary Ceph gets monitors on both sides of the link. Secondary Ceph only has monitors on the secondary side. > > In a netsplit situation, the primary Ceph will maintain quorum with both nodes on the primary side. The secondary Ceph cluster only exists separately for this netsplit situation and the secondary k8s cluster can continue unaffected. > > With this in place, the primary side can continue operating with either primary node downed for maintenance via a suboptimal quorum over the WAN link. I cannot do that today. > > I am sacrificing the case where there is a netsplit at the same time I am doing maintenance. > > Thanks for your input! > Brian > >

1 0

RGWReshardLock::lock failed to acquire lock ret=-16
by Josh Haft 12 Mar '20

12 Mar '20

Hi, Currently running Mimic 13.2.5. We had reports this morning of timeouts and failures with PUT and GET requests to our Ceph RGW cluster. I found these messages in the RGW log: RGWReshardLock::lock failed to acquire lock on bucket_name:bucket_instance ret=-16 NOTICE: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is still resharding, please retry Which were preceded by many of these, which I think are normal/expected. check_bucket_shards: resharding needed: stats.num_objects=6415879 shard max_objects=6400000 Our RGW cluster sits behind haproxy which notified me approx 90 seconds after the first 'resharding needed' message that no backends were available. It appears this dynamic reshard process caused the RGWs to lock up for a period of time. Roughly 2 minutes later the reshard error messages stop and operation returns to normal. Looking back through previous RGW logs, I see a similar event from about a week ago, on the same bucket. We have several buckets with shard counts exceeding 1k (this one only has 128), and much larger object counts, so clearly this isn't the first time dynamic sharding has been invoked on this cluster. Has anyone seen this? I expect it will come up again, and can turn up debugging if that'll help. Thanks for any assistance! Josh

1 1

ceph-mon store.db disk usage increase on OSD-Host fail
by Hartwig Hauschild 12 Mar '20

12 Mar '20

Hi, I'm (still) testing upgrading from Luminous to Nautilus and ran into the following situation: The lab-setup I'm testing in has three OSD-Hosts. If one of those hosts dies the store.db in /var/lib/ceph/mon/ on all my Mon-Nodes starts to rapidly grow in size until either the OSD-host comes back up or disks are full. On another cluster that's still on Luminous I don't see any growth at all. Is that a difference in behaviour between Luminous and Nautilus or is that caused by the lab-setup only having three hosts and one lost host causing all PGs to be degraded at the same time? -- Cheers, Hardy

3 7

Single machine / multiple monitors
by Brian Topping 12 Mar '20

12 Mar '20

Hi, I’m getting conflicting reads from the documentation. It seems that by using the “cluster name”[1], multiple clusters can be run in parallel on the same hardware. In trying to set this up with `ceph-deploy`, I see the man page[2] says "if it finds the distro.init to be sysvinit (Fedora, CentOS/RHEL etc), it doesn't allow installation with custom cluster name and uses the default name ceph for the cluster”. Is it possible to run multiple clusters on the same hardware with CentOS 7 as the base OS? Thanks, Brian [1] https://docs.ceph.com/docs/nautilus/install/manual-deployment/#monitor-boot… [2] https://docs.ceph.com/docs/nautilus/man/8/ceph-deploy/?highlight=ceph-deplo…

2 2

Rados example: create namespace, user for this namespace, read and write objects with created namespace and user
by Rodrigo Severo - Fábrica 11 Mar '20

11 Mar '20

Hi, I'm trying to create a namespace in rados, create a user that has access to this created namespace and with rados command line utility read and write objects in this created namespace using the created user. I can't find an example on how to do it. Can someone point me to such example or show me how to do it? Regards, Rodrigo Severo

3 3

Setting user in rados command line utility
by Rodrigo Severo - Fábrica 11 Mar '20

11 Mar '20

Hi, How can I set the user to be used by rados command line utility? I can see no option for that in https://docs.ceph.com/docs/nautilus/man/8/rados/ Regards, Rodrigo Severo

2 2

Re: Nautilus OSD memory consumption?
by Nigel Williams 11 Mar '20

11 Mar '20

On Thu, 27 Feb 2020 at 06:27, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > If the heap stats reported by telling the OSD `heap stats` is large, telling each `heap release` may be useful. I suspect a TCMALLOC shortcoming. osd.158 tcmalloc heap stats:------------------------------------------------ MALLOC: 5722761448 ( 5457.7 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 311621760 ( 297.2 MiB) Bytes in central cache freelist MALLOC: + 26242992 ( 25.0 MiB) Bytes in transfer cache freelist MALLOC: + 62721768 ( 59.8 MiB) Bytes in thread cache freelists MALLOC: + 113340608 ( 108.1 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 6236688576 ( 5947.8 MiB) Actual memory used (physical + swap) MALLOC: + 21415870464 (20423.8 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 27652559040 (26371.5 MiB) Virtual address space used MALLOC: MALLOC: 394518 Spans in use MALLOC: 37 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory.

2 2

Accidentally removed client.admin caps - fix via mon doesn't work
by wittler＠appvisory.com 11 Mar '20

11 Mar '20

Hello Guys, Unfortunately, I''ve deleted some caps from client.admin and tried the following solution so set them back: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015474.html I’ve tried the following: # ssh’d to a mon node and changed dir to the mon directory cd /var/lib/ceph/mon/<monname> # tried to authenticate with the monitor keyring and set the client.admin caps to give back full permissions ceph -n mon.<monid> --keyring keyring auth caps client.admin mds 'allow *' osd 'allow *' mon 'allow *' When i try to modify the client.admin caps, the command just hangs at the shell until i press ctrl-c, which gets aknowledged with "Cluster connection aborted” Also i can’t see the the connection in the ceph.audit.log Do i make a mistake here or is this "workaround" not supported anymore in nautilus (14.2.6)? Regards

3 4

Re: cephfs snap mkdir strange timestamp
by Marc Roos 11 Mar '20

11 Mar '20

Sorry I am not really developer nor now details about POSIX/fs's. But if I may ask, why are you getting this time from the parent? -----Original Message----- Sent: 11 March 2020 11:32 To: Jeff Layton Cc: Marc Roos; ceph-users; ceph-devel(a)vger.kernel.org Subject: Re: [ceph-users] cephfs snap mkdir strange timestamp On Tue, Mar 10, 2020 at 01:39:29PM -0400, Jeff Layton wrote: ... > > Signed-off-by: Luis Henriques <lhenriques(a)suse.com> > > --- > > fs/ceph/inode.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c index > > d01710a16a4a..f4e78ade0871 100644 > > --- a/fs/ceph/inode.c > > +++ b/fs/ceph/inode.c > > @@ -82,6 +82,8 @@ struct inode *ceph_get_snapdir(struct inode *parent) > > inode->i_mode = parent->i_mode; > > inode->i_uid = parent->i_uid; > > inode->i_gid = parent->i_gid; > > + inode->i_mtime = parent->i_mtime; > > + inode->i_ctime = parent->i_ctime; > > inode->i_op = &ceph_snapdir_iops; > > inode->i_fop = &ceph_snapdir_fops; > > ci->i_snap_caps = CEPH_CAP_PIN; /* so we can open */ > > What about the atime, and the ci->i_btime ? Yeah, probably makes sense too, although the fuse client doesn't seem to touch atime (it does change btime, I missed that). I'll send v2 in a bit. Cheers, -- Luis

1 0

Bucket notification with kafka error
by 曹海旺 11 Mar '20

11 Mar '20

HI, I'm sorry to bother you again I want to use kafka to queue the notifications , I add a topic named kafka,and put the notification config xml The topic info: <ListTopicsResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> <ListTopicsResult> <Topics> <member> <User>sr</User> <Name>kafka</Name> <EndPoint> <EndpointAddress>kafka://192.168.3.250:9092</EndpointAddress> <EndpointArgs>kafka-ack-level=broker&push-endpoint=kafka://192.168.3.250:9092</EndpointArgs> <EndpointTopic>kafka</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:default::kafka</TopicArn> </member> <member> <User>sr</User> <Name>kafka_kafka</Name> <EndPoint> <EndpointAddress>kafka://192.168.3.250:9092</EndpointAddress> <EndpointArgs>kafka-ack-level=broker&push-endpoint=kafka://192.168.3.250:9092</EndpointArgs> <EndpointTopic>kafka</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:default::kafka</TopicArn> </member> <member> <User>sr</User> <Name>webno</Name> <EndPoint> <EndpointAddress>http://192.168.1.114:8080/s3/sn</EndpointAddress> <EndpointArgs>push-endpoint=http://192.168.1.114:8080/s3/sn</EndpointArgs> <EndpointTopic>webno</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:default::webno</TopicArn> </member> </Topics> </ListTopicsResult> <ResponseMetadata> <RequestId>c4b84c5b-1e88-4f16-9863-7f68872d91a4.744394.135</RequestId> </ResponseMetadata> </ListTopicsResponse> and the put notification body : <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <TopicConfiguration> <Id>kafka</Id> <Topic>arn:aws:sns:default::kafka</Topic> </TopicConfiguration> </NotificationConfiguration> The web notification works fine ,but when I use the kafka(version 1.0 jdk 1.8) I got the debug info : 020-03-11 12:46:38.612 7fd81eeb1700 20 get_system_obj_state: s->obj_tag was set empty 2020-03-11 12:46:38.612 7fd81eeb1700 10 cache get: name=default.rgw.log++pubsub.user.sr.bucket.osstest/c4b84c5b-1e88-4f16-9863-7f68872d91a4.14175.1 : hit (requested=0x1, cached=0x17) 2020-03-11 12:46:38.612 7fd81eeb1700 20 notification: 'kafka' on topic: 'kafka' and bucket: 'osstest' (unique topic: 'kafka_kafka') apply to event of type: 's3:ObjectCreated:Put' 2020-03-11 12:46:38.612 7fd81eeb1700 1 ERROR: failed to create push endpoint: kafka://192.168.3.250:9092 due to: pubsub endpoint configuration error: unknown schema in: kafka://192.168.3.250:9092 2020-03-11 12:46:38.612 7fd81eeb1700 5 req 126 0.186s s3:put_obj WARNING: publishing notification failed, with error: -22 2020-03-11 12:46:38.612 7fd81eeb1700 2

2 1

reset pgs not deep-scrubbed in time
by Stefan Priebe - Profihost AG 11 Mar '20

11 Mar '20

Hello, is there any way to reset deep-scrubbed time for pgs? The cluster was accidently in state nodeep-scrub and is now unable to deep scrub fast enough. Is there any way to force mark all pgs as deep scrubbed to start from 0 again? Greets, Stefan

2 1

rbd-mirror replay is very slow - but initial bootstrap is fast
by Ml Ml 10 Mar '20

10 Mar '20

Hello List, when i initially enable journal/mirror on an image it gets bootstrapped to my site-b pretty quickly with 250MB/sec which is about the IO Write limit. Once its up2date, the replay is very slow. About 15KB/sec and the entries_behind_maste is just running away: root@ceph01:~# rbd --cluster backup mirror pool status rbd-cluster6 --verbose health: OK images: 3 total 3 replaying ... vm-112-disk-0: global_id: 60a795c3-9f5d-4be3-b9bd-3df971e531fa state: up+replaying description: replaying, master_position=[object_number=623, tag_tid=3, entry_tid=345567], mirror_position=[object_number=35, tag_tid=3, entry_tid=18371], entries_behind_master=327196 last_update: 2020-03-10 11:36:44 ... Write traffic on the source is about 20/25MB/sec. On the Source i run 14.2.6 and on the destination 12.2.13. Any idea why the replaying is sooo slow? Thanks, Michael

3 8

FW: Warning: could not send message for past 4 hours
by Marc Roos 10 Mar '20

10 Mar '20

-----Original Message----- From: Mail Delivery Subsystem [mailto:MAILER-DAEMON] Sent: 10 March 2020 19:01 Subject: Warning: could not send message for past 4 hours ********************************************** ** THIS IS A WARNING MESSAGE ONLY ** ** YOU DO NOT NEED TO RESEND YOUR MESSAGE ** ********************************************** The original message was received at Tue, 10 Mar 2020 15:00:07 +0100 from localhost.localdomain [127.0.0.1] ----- Transcript of session follows ----- 451 croit.io: Name server timeout Warning: message still undelivered after 4 hours Will keep trying until message is 5 days old

1 0

Re: cephfs snap mkdir strange timestamp
by Paul Emmerich 10 Mar '20

10 Mar '20

There's an xattr for this: ceph.snap.btime IIRC Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Mar 10, 2020 at 11:42 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > If I make a directory in linux the directory has the date of now, why is > this not with creating a snap dir? Is this not a bug? One expects this > to be the same as in linux not???? > > [ @ test]$ mkdir temp > > [ @os0 test]$ ls -arltn > total 28 > drwxrwxrwt. 27 0 0 20480 Mar 10 11:38 .. > drwxrwxr-x 2 801 801 4096 Mar 10 11:38 temp > drwxrwxr-x 3 801 801 4096 Mar 10 11:38 . > > > [ @ .snap]# mkdir test > [ @ .snap]# ls -lartn > total 0 > drwxr-xr-x 861886554 0 0 8390344070786420358 Jan 1 1970 . > drwxr-xr-x 4 0 0 2 Mar 6 14:43 test > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-9 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-8 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-7 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-6 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-5 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-10 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 .. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

ceph: Can't lookup inode 1 (err: -13)
by Marc Roos 10 Mar '20

10 Mar '20

For testing purposes I changed the kernel 3.10 for a 5.5, now I am getting these messages. I assume the 3.10 was just never displaying these. Could this be a problem with my caps of the fs id user? [Mon Mar 9 23:10:52 2020] ceph: Can't lookup inode 1 (err: -13) [Mon Mar 9 23:12:03 2020] ceph: Can't lookup inode 1 (err: -13) [Mon Mar 9 23:13:12 2020] ceph: Can't lookup inode 1 (err: -13) [Mon Mar 9 23:14:19 2020] ceph: Can't lookup inode 1 (err: -13)

2 4

Nautilus cephfs usage
by Yoann Moulin 10 Mar '20

10 Mar '20

Hello, I have a Nautilus cluster with a cephfs volume, on grafana, it shows that cephfs_data pool is almost full[1] but if I give a look to the pool usage, it looks like I have plenty of space. Which metrics are used by grafana? 1. https://framapic.org/5r7J86s55x6k/jGSIsjEUPYMU.png pool usage: > artemis@icitsrv5:~$ ceph df detail > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 662 TiB 296 TiB 366 TiB 366 TiB 55.32 > TOTAL 662 TiB 296 TiB 366 TiB 366 TiB 55.32 > > POOLS: > POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR > .rgw.root 3 8.1 KiB 15 2.8 MiB 0 63 TiB N/A N/A 15 0 B 0 B > default.rgw.control 4 0 B 8 0 B 0 63 TiB N/A N/A 8 0 B 0 B > default.rgw.meta 5 26 KiB 85 16 MiB 0 63 TiB N/A N/A 85 0 B 0 B > default.rgw.log 6 0 B 207 0 B 0 63 TiB N/A N/A 207 0 B 0 B > cephfs_data 7 113 TiB 139.34M 186 TiB 49.47 138 TiB N/A N/A 139.34M 0 B 0 B > cephfs_metadata 8 54 GiB 10.21M 57 GiB 0.03 63 TiB N/A N/A 10.21M 0 B 0 B > default.rgw.buckets.data 9 122 TiB 54.57M 173 TiB 47.70 138 TiB N/A N/A 54.57M 0 B 0 B > default.rgw.buckets.index 10 2.6 GiB 19.97k 2.6 GiB 0 63 TiB N/A N/A 19.97k 0 B 0 B > default.rgw.buckets.non-ec 11 67 MiB 186 102 MiB 0 63 TiB N/A N/A 186 0 B 0 B > device_health_metrics 12 1.2 MiB 145 1.2 MiB 0 63 TiB N/A N/A 145 0 B 0 B Best, -- Yoann Moulin EPFL IC-IT

1 0

Radosgw dynamic sharding jewel -> luminous
by Robert LeBlanc 10 Mar '20

10 Mar '20

I just upgraded a cluster that I inherited from Jewel to Luminous and trying to work through the new warnings/errors. I got the message about 3 OMAP objects being too big, all of them in the default.rgw.buckets.index pool. I expected that dynamic sharding should kick in, but no luck after several days. I looked at $ radosgw-admin reshard list [2020-03-02 04:27:22.303601 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000000 2020-03-02 04:27:22.305403 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000001 2020-03-02 04:27:22.307038 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000002 2020-03-02 04:27:22.317932 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000003 2020-03-02 04:27:22.348383 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000004 2020-03-02 04:27:22.349212 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000005 2020-03-02 04:27:22.349853 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000006 2020-03-02 04:27:22.350490 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000007 2020-03-02 04:27:22.351256 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000008 2020-03-02 04:27:22.351843 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000009 2020-03-02 04:27:22.353225 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000010 2020-03-02 04:27:22.353910 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000011 2020-03-02 04:27:22.367161 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000012 2020-03-02 04:27:22.367741 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000013 2020-03-02 04:27:22.368498 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000014 ] And searching on the Internet indicates tha Luminous added a new "reshard" namespace that the radosgw user needs access to. I'm not sure which pool this namespace was added to (being from Jewel there are a slew of rgw pools) and I'm not sure which rados-gw user it's talking about. I can't find a keyring for rados-admin, but it works so I assume that it is using the admin key ring. The permissions are open. I even appended the namespace option to the admin caps like follows: [client.admin] key = SECRET caps mds = "allow *" caps mon = "allow *" caps: [osd] allow * namespace="*" But I get a new error: $ radosgw-admin reshard list 2020-03-02 05:31:40.875642 7fdb983f7e40 0 failed reading realm info: ret -1 (1) Operation not permitted Any nudge in the right direction would be helpful. I manually sharded the indexes, but I'd really like to have it done automatically from now on so I don't have to worry about it. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 4

Monitors' election failed on VMs : e4 handle_auth_request failed to assign global_id
by Yoann Moulin 10 Mar '20

10 Mar '20

Hello, On a Nautilus cluster, I'd like to move monitors from bare metal servers to VMs to prepare a migration. I have added 3 new monitors on 3 VMs and I'd like to stop the 3 old monitors daemon. But I soon as I stop the 3rd old monitor, the cluster stuck because the election of a new monitor fails. The 3 old monitors are in 14.2.4-1xenial The 3 new monitors are in 14.2.7-1bionic > 2020-03-09 16:06:00.167 7fc4a3138700 1 mon.icvm0017(a)3(peon).paxos(paxos active c 20918592..20919120) lease_timeout -- calling new election > 2020-03-09 16:06:02.143 7fc49f931700 1 mon.icvm0017@3(probing) e4 handle_auth_request failed to assign global_id Did I miss something? In attachment : some logs and ceph.conf Thanks for your help. Best, -- Yoann Moulin EPFL IC-IT

3 3

Clear health warning
by Peter Eisch 09 Mar '20

09 Mar '20

Hi, (nautilus, 14.2.8, whole cluster) I doodled with adding a second cephfs and the project got canceled. I removed the unused cephfs with "ceph fs rm dream --yes-i-really-mean-it" and that worked as expected. I have a lingering health warning though which won't clear. The original cephfs1 volume exists and is healthy: [root@cephmon-03]# ceph fs ls name: cephfs1, metadata pool: stp.cephfs_metadata, data pools: [stp.cephfs_data ] [root@cephmon-03]# ceph mds stat cephfs1:3 {0=cephmon-03=up:active,1=cephmon-02=up:active,2=cephmon-01=up:active} [root@cephmon-03]# ceph health detail HEALTH_WARN insufficient standby MDS daemons available MDS_INSUFFICIENT_STANDBY insufficient standby MDS daemons available have 0; want 1 more [root@cephmon-03]# I have not yet deleted the pools for 'dream', the second cephfs definition. (There is nothing in it.) Before deleting the pools is there a command to clear this warning? Thanks! peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.64

2 4

Link to Nautilus upgrade
by Peter Eisch 09 Mar '20

09 Mar '20

Hi, When upgrading a cluster from Luminous to Nautilus I followed a page on ceph.com. I need to do another cluster and while I tagged the link, the page no longer exists. https://docs.ceph.com/master/releases/nautilus/#nautilus-old-upgrade Might anyone have either an updated link or point me to how to find the contents of best steps? Thanks, peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.64

2 2

Hardware feedback before purchasing for a PoC
by Ignacio Ocampo 09 Mar '20

09 Mar '20

Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase: The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*. System configuration: https://pcpartpicker.com/list/cfDpDx ($400 USD per node) Some notes about the configuration: - 4-core processor for 4 OSD daemons - 8GB RAM for the first 4TB of storage, that will increase to 16GB of RAM when 16TB of storage. - Motherboard: - 4 x SATA 6 Gb/s (one per each OSD disk) - 2 x PCI-E x1 Slots (1 will be used for an additional Gigabit Ethernet) - 1 x M.2 Slots for the host OS - Ram can increase up-to 32 GB, and another SATA 6b/s controller can be added on PCI-E x1 for growth up to *32TB* As noted, the plan is to deploy nodes with *4TB* and gradually add *12TB* as needed, memory also should be increased to *16GB* after *8TB* threshold. Edit Questions to validate before the purchase 1. Does the hardware components make sense for the *16TB* growth projection? 2. Is it easy to gradually add more capacity to each node (*4TB* each time per node)? Thanks for your support! -- Ignacio Ocampo

4 5

A fast tool to export/copy a pool
by Vladimir Brik 09 Mar '20

09 Mar '20

I am wondering if there exists a tool, faster than "rados export", that can copy and restore read-only pools (to/from another pool or file system). It looks like "rados export" is very slow because it is single threaded (the best I can tell, --workers doesn't make a difference). Vlad

1 0

ceph df hangs
by Rebecca CH 09 Mar '20

09 Mar '20

Hello When I run 'ceph df' on one of osd nodes, the command get hanged and no output to the screen. How to resolve it? Thanks.

2 1

Re: ceph rbd volumes/images IO details
by M Ranga Swami Reddy 09 Mar '20

09 Mar '20

On Fri, Mar 6, 2020 at 1:06 AM M Ranga Swami Reddy <swamireddy(a)gmail.com> wrote: > Hello, > Can we get the IOPs of any rbd image/volume? > > For ex: I have created volumes via OpenStack Cinder. Want to know > the IOPs of these volumes. > > In general - we can get pool stats, but not seen the per volumes stats. > > Any hint here? Appreciated. >

3 3

Ceph (version 14.2.7) RGW STS AccessDenied
by 曹海旺 09 Mar '20

09 Mar '20

Hi ,I want to use the sts to get a temporary credentials,but I got 403 . The user info : { "user_id": "sr", "display_name": "shiran", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "sr", "access_key": "FBGYU7CUBUACP2OYBE76", "secret_key": "RoovZBBNdlOvVS4LzuuuCsZ1QPA6W81mbNXvonYD" } ], "swift_keys": [], "caps": [ { "type": "roles", "perm": "*" }, { "type": "user-policy", "perm": "*" } ], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } and the role info : { "id": "c983c0df-a49c-4e36-957d-7d709f65e292", "name": "S3Access", "path": "/application_abc/component_xyz/", "arn": "arn:aws:iam:::role/application_abc/component_xyz/S3Access", "create_date": "2020-03-04T08:19:41.24Z", "max_session_duration": 3600, "assume_role_policy_document": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/sr\"]},\"Action\":[\"sts:AssumeRole\"]}]}" }, I use AWA java sdk to AssumeRole ,code like: public static final String ACCESSKEY_STRING = “xxxxxxxxxxxxxx"; public static final String BUCKETNAME_STRING = "osstest"; public static final String SECRETKEY_STRING = “xxxxxxxxxxxxxxxxxxxxx"; public static final String ENDPOINT_STRING = "http://192.168.3.250”;// haproxy String clientRegion = ENDPOINT_STRING; String roleARN = "arn:aws:iam:::role/application_abc/component_xyz/S3Access"; String roleSessionName = "bob"; String bucketName = ENDPOINT_STRING; AWSCredentials credentials = new BasicAWSCredentials(ACCESSKEY_STRING, SECRETKEY_STRING); ClientConfiguration clientConfig = new ClientConfiguration(); clientConfig.setProtocol(Protocol.HTTP); clientConfig.setSignerOverride("AWS3SignerType"); AWSSecurityTokenService sts_client = AWSSecurityTokenServiceClientBuilder.standard() .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(ENDPOINT_STRING, "")) .withCredentials(new AWSStaticCredentialsProvider(credentials)).withClientConfiguration(clientConfig) .build(); GetSessionTokenRequest session_token_request = new GetSessionTokenRequest(); AssumeRoleRequest roleRequest = new AssumeRoleRequest() .withRoleArn(roleARN) .withRoleSessionName(roleSessionName).withTokenCode("sts"); session_token_request.setDurationSeconds(3600); AssumeRoleResult roleResponse = sts_client.assumeRole(roleRequest); Credentials sessionCredentials = roleResponse.getCredentials(); I got the response: <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx000000000000000000016-005e64fa9e-b5b47-default</RequestId><HostId>b5b47-default-default</HostId></Error>" The debug info in the log file like this : 1 ====== starting new request req=0x5592abefc910 ===== 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s initializing for trans_id = tx000000000000000000016-005e64fa9e-b5b47-default 2020-03-08 22:01:02.984 7f2548f95700 10 rgw api priority: s3=7 s3website=6 2020-03-08 22:01:02.984 7f2548f95700 10 host=rgw1 2020-03-08 22:01:02.984 7f2548f95700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2020-03-08 22:01:02.984 7f2548f95700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ 2020-03-08 22:01:02.984 7f2548f95700 10 meta>> HTTP_X_AMZ_DATE 2020-03-08 22:01:02.984 7f2548f95700 10 meta>> HTTP_X_AMZN_AUTHORIZATION 2020-03-08 22:01:02.984 7f2548f95700 10 x>> x-amz-date:Sun, 08 Mar 2020 14:01:02 GMT 2020-03-08 22:01:02.984 7f2548f95700 10 x>> x-amzn-authorization:AWS3 AWSAccessKeyId=FBGYU7CUBUACP2OYBE76,Algorithm=HmacSHA256,SignedHeaders=Host;X-Amz-Date,Signature=D7p74AFsJn4J823e6Up4Zo2eRV/n5PFmyQ2/w+kOaqY= 2020-03-08 22:01:02.984 7f2548f95700 20 get_handler handler=26RGWHandler_REST_Service_S3 2020-03-08 22:01:02.984 7f2548f95700 10 handler=26RGWHandler_REST_Service_S3 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s getting op 4 2020-03-08 22:01:02.984 7f2548f95700 10 Content of POST: Action=AssumeRole&Version=2011-06-15&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2Fapplication_abc%2Fcomponent_xyz%2FS3Access&RoleSessionName=bob&SerialNumber=sdfsdf&TokenCode=sts 2020-03-08 22:01:02.984 7f2548f95700 10 Key: ActionValue: AssumeRole 2020-03-08 22:01:02.984 7f2548f95700 10 Key: VersionValue: 2011-06-15 2020-03-08 22:01:02.984 7f2548f95700 10 Key: RoleArnValue: arn:aws:iam:::role/application_abc/component_xyz/S3Access 2020-03-08 22:01:02.984 7f2548f95700 10 Key: RoleSessionNameValue: bob 2020-03-08 22:01:02.984 7f2548f95700 10 Key: SerialNumberValue: sdfsdf 2020-03-08 22:01:02.984 7f2548f95700 10 Key: TokenCodeValue: sts 2020-03-08 22:01:02.984 7f2548f95700 10 req 22 0.000s sts:assume_role scheduling with dmclock client=0 cost=1 2020-03-08 22:01:02.984 7f2548f95700 10 op=16RGWSTSAssumeRole 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s sts:assume_role verifying requester 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::s3::S3AnonymousEngine granted access 2020-03-08 22:01:02.984 7f2548f95700 20 req 22 0.000s sts:assume_role rgw::auth::s3::AWSAuthStrategy granted access 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s sts:assume_role normalizing buckets and tenants 2020-03-08 22:01:02.984 7f2548f95700 10 s->object=<NULL> s->bucket= 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s sts:assume_role init permissions 2020-03-08 22:01:02.985 7f2548f95700 20 get_system_obj_state: rctx=0x5592abefb400 obj=default.rgw.meta:users.uid:anonymous state=0x5592aa8d99c0 s->prefetch_data=0 2020-03-08 22:01:02.985 7f2548f95700 10 cache get: name=default.rgw.meta+users.uid+anonymous : type miss (requested=0x6, cached=0x0) 2020-03-08 22:01:02.985 7f2548f95700 20 WARNING: blocking librados call 2020-03-08 22:01:02.985 7f2548f95700 1 -- 192.168.3.247:0/1491423941 --> [v2:192.168.3.244:6800/2853,v1:192.168.3.244:6801/2853] -- osd_op(unknown.0.0:159377 3.0 3:0ab27964:users.uid::anonymous:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e282) v8 -- 0x5592abea3b80 con 0x5592ab6acc00 2020-03-08 22:01:02.985 7f2593ab4700 1 -- 192.168.3.247:0/1491423941 <== osd.5 v2:192.168.3.244:6800/2853 4 ==== osd_op_reply(159377 anonymous [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 195+0+0 (crc 0 0 0) 0x5592abda1400 con 0x5592ab6acc00 2020-03-08 22:01:02.985 7f2548f95700 10 cache put: name=default.rgw.meta+users.uid+anonymous info.flags=0x0 2020-03-08 22:01:02.985 7f2548f95700 10 moving default.rgw.meta+users.uid+anonymous to cache LRU end 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role recalculating target 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role reading permissions 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role init op 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role verifying op mask 2020-03-08 22:01:02.985 7f2548f95700 20 req 22 0.001s sts:assume_role required_mask= 0 user.op_mask=7 2020-03-08 22:01:02.985 7f2548f95700 2 req 22 0.001s sts:assume_role verifying op permissions 2020-03-08 22:01:02.985 7f2548f95700 20 get_system_obj_state: rctx=0x5592abefb7f0 obj=default.rgw.meta:roles:role_names.S3Access state=0x5592aa8d99c0 s->prefetch_data=0 2020-03-08 22:01:02.985 7f2548f95700 10 cache get: name=default.rgw.meta+roles+role_names.S3Access : type miss (requested=0x6, cached=0x3) 2020-03-08 22:01:02.985 7f2548f95700 20 WARNING: blocking librados call 2020-03-08 22:01:02.985 7f2548f95700 1 -- 192.168.3.247:0/1491423941 --> [v2:192.168.3.245:6808/2863,v1:192.168.3.245:6809/2863] -- osd_op(unknown.0.0:159378 3.3 3:c86e03bb:roles::role_names.S3Access:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e282) v8 -- 0x5592abf18000 con 0x5592ab6ad400 2020-03-08 22:01:02.986 7f2593ab4700 1 -- 192.168.3.247:0/1491423941 <== osd.8 v2:192.168.3.245:6808/2863 3 ==== osd_op_reply(159378 role_names.S3Access [getxattrs,stat] v0'0 uv4 ondisk = 0) v8 ==== 205+0+20 (crc 0 0 0) 0x5592abda1400 con 0x5592ab6ad400 2020-03-08 22:01:02.986 7f2548f95700 10 cache put: name=default.rgw.meta+roles+role_names.S3Access info.flags=0x6 2020-03-08 22:01:02.986 7f2548f95700 10 moving default.rgw.meta+roles+role_names.S3Access to cache LRU end 2020-03-08 22:01:02.986 7f2548f95700 20 get_system_obj_state: s->obj_tag was set empty 2020-03-08 22:01:02.986 7f2548f95700 10 cache get: name=default.rgw.meta+roles+role_names.S3Access : hit (requested=0x1, cached=0x7) 2020-03-08 22:01:02.986 7f2548f95700 20 get_system_obj_state: rctx=0x5592abefb7f0 obj=default.rgw.meta:roles:roles.c983c0df-a49c-4e36-957d-7d709f65e292 state=0x5592aa8d99c0 s->prefetch_data=0 2020-03-08 22:01:02.986 7f2548f95700 10 cache get: name=default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 : type miss (requested=0x6, cached=0x3) 2020-03-08 22:01:02.986 7f2548f95700 20 WARNING: blocking librados call 2020-03-08 22:01:02.986 7f2548f95700 1 -- 192.168.3.247:0/1491423941 --> [v2:192.168.3.244:6800/2853,v1:192.168.3.244:6801/2853] -- osd_op(unknown.0.0:159379 3.0 3:0e42bcf0:roles::roles.c983c0df-a49c-4e36-957d-7d709f65e292:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e282) v8 -- 0x5592abf182c0 con 0x5592ab6acc00 2020-03-08 22:01:02.986 7f2593ab4700 1 -- 192.168.3.247:0/1491423941 <== osd.5 v2:192.168.3.244:6800/2853 5 ==== osd_op_reply(159379 roles.c983c0df-a49c-4e36-957d-7d709f65e292 [getxattrs,stat] v0'0 uv4 ondisk = 0) v8 ==== 228+0+20 (crc 0 0 0) 0x5592abda1400 con 0x5592ab6acc00 2020-03-08 22:01:02.986 7f2548f95700 10 cache put: name=default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 info.flags=0x6 2020-03-08 22:01:02.986 7f2548f95700 10 moving default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 to cache LRU end 2020-03-08 22:01:02.986 7f2548f95700 20 get_system_obj_state: s->obj_tag was set empty 2020-03-08 22:01:02.986 7f2548f95700 10 cache get: name=default.rgw.meta+roles+roles.c983c0df-a49c-4e36-957d-7d709f65e292 : hit (requested=0x1, cached=0x7) 2020-03-08 22:01:02.986 7f2548f95700 20 op->ERRORHANDLER: err_no=-1 new_err_no=-1 2020-03-08 22:01:02.986 7f2548f95700 2 req 22 0.002s sts:assume_role op status=0 2020-03-08 22:01:02.986 7f2548f95700 2 req 22 0.002s sts:assume_role http status=403

2 1

RGW jaegerTracing
by Abhinav Singh 09 Mar '20

09 Mar '20

I am trying to implement jaeger tracing in RGW, I need some advice regarding on which functions should I actually tracing to get a good actual performance status of clusters Till now I am able to deduce followings : 1.I think we need to provide tracing functions where the `rgw` is communicating with the librados, (particularly the librgw where the communication is actually happening), because http request and response not to be considered for tracing because that depends on clients internet speed. 2.In librgw the functions like this here <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> and its corresponding overloading methods and also the this function here <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> and its corresponding overloaded functions. 3.I see that pools are ultimately used to enter the crush algorithm for writing data, so I think the ceation of pools should also be taken into account while tracing,(creation of pool should be main span and these functions <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> should be its child span). Functionality of buckets like that of this <https://github.com/ceph/ceph/blob/0360bea127397a41eb282a1eef9af4ff4477b9d4/…> do not require tracing beacuse they are http requests. Any kind of guidance will be of great help. Thank You.

2 1

log_latency_fn slow operation
by 徐蕴 08 Mar '20

08 Mar '20

Hi, Our cluster (14.2.6) has sporadic slow ops warnings since upgrading from Jewel 1 month ago. Today I checked the OSD log files and found out a lot of entries like: ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.16871s, txc = 0x55e33ae41b80 ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.15158s, txc = 0x55e3639b3340 ceph-osd.5.log:2020-03-04 10:33:31.592 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.77361s, txc = 0x55e3379cc840 ceph-osd.5.log:2020-03-04 10:33:52.666 7f18ca41f700 0 bluestore(/var/lib/ceph/osd/ceph-5) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.42519s, txc = 0x55e33722d600 or /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 00:41:31.110 7f3dc0bc8700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for submit_transact, latency = 8.1279s /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 00:41:31.110 7f3dd1bea700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for kv_final, latency = 7.88786s /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 02:21:35.180 7f3dd1bea700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for kv_final, latency = 6.06171s /var/log/kolla/ceph/ceph-osd.7.log:2020-03-04 05:31:30.298 7f3dc1bca700 0 bluestore(/var/lib/ceph/osd/ceph-7) log_latency slow operation observed for submit_transact, latency = 5.34228s The cluster setup is: SATA SSD (as DB) + SATA HDD 1:3. Any suggest how to debug this problem? Thank you! br, Xu Yun

2 1

Welcome to the "ceph-users" mailing list
by Abhinav Singh 08 Mar '20

08 Mar '20

singhabhinav9051571833(a)gmail.com

1 0

Disabling Telemetry
by m＠silvenga.com 07 Mar '20

07 Mar '20

Is there another way to disable telemetry then using: > ceph telemetry off > Error EIO: Module 'telemetry' has experienced an error and cannot handle commands: cannot concatenate 'str' and 'UUID' objects I'm attempting to get all my clusters out of a constant HEALTH_ERR state caused by either the above error or the telemetry endpoint being down.

2 2

High CPU usage by ceph-mgr in 14.2.6
by jbardgett＠godaddy.com 07 Mar '20

07 Mar '20

After upgrading one of our clusters from Luminous 12.2.12 to Nautilus 14.2.6, I am seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). The way we found this was due to Prometheus being unable to report out certain pieces of data, specifically OSD Usage, OSD Apply and Commit Latency. Which are all similar issues people were having in previous versions of Nautilus. Bryan Stillwell reported this previously on a separate cluster, 14.2.5, we have here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VW3GNVJGOOW… That issue was resolved with the upgrade to 14.2.6. We are seeing a similar issue on this other cluster with a couple differences. This cluster has 1900+ OSD in it, the previous one had 300+ The top user is libceph-common, instead of mmap 4.86% libceph-common.so.0 [.] EventCenter::create_time_event 2.78% [kernel] [k] nmi 2.64% libstdc++.so.6.0.19 [.] __dynamic_cast On all our other clusters that have been upgraded to 14.2.6 we are not experiencing this issue, the next largest being 800+ OSD. We feel this is related to the size of the cluster, similarly to the previous report. Anyone else experiencing this and/or can provide some direction on how to go about resolving this? Thanks, Joe

6 7

MDS Issues
by DHilsbos＠performair.com 07 Mar '20

07 Mar '20

All; We are in the middle of upgrading our primary cluster from 14.2.5 to 14.2.8. Our cluster utilizes 6 MDSs for 3 CephFS file systems. 3 MDSs are collocated with MON/MGR, and 3 MDSs are collocated with OSDs. At this point we have upgraded all 3 of the MON/MDS/MGR servers. The MDS on 2 of the 3 is currently not working, and we are seeing the below log messages. 2020-03-06 11:12:56.184 <> -1 mds.<daemon> unable to obtain rotating service keys; retrying 2020-03-06 11:13:26.184 <> 0 monclient: wait_auth_rotating timed out after 30 2020-03-06 11:13:26.184 <> -1 mds.<daemon> ERROR: failed to refresh rotating keys, maximum retry time reached. 2020-03-06 11:13:26.184 <> 1 mds.<daemon> suicide! Wanted state up:boot Any ideas? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 2

Ceph Performance of Micron 5210 SATA?
by Hermann Himmelbauer 06 Mar '20

06 Mar '20

Hi, Does someone know if the following harddisk has a decent performance in a ceph cluster: Micron 5210 ION 1.92TB, SATA (MTFDDAK1T9QDE-2AV1ZABYY) The spec state, that the disk has power loss protection, however, I'd nevertheless like to make sure that all goes well with this disk. Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

5 7

How to get the size of cephfs snapshot?
by Marc Roos 06 Mar '20

06 Mar '20

If I do getfattr --only-values --absolute-names -d -m ceph.dir.rbytes .snap/snap-6 then I am getting the same size as the unaltered source. This needs to be 0 not? And when the source is changing, this should increase? Or do I not understand correctly?

2 1

pg_num as power of two adjustment: only downwards?
by Rodrigo Severo - Fábrica 06 Mar '20

06 Mar '20

Hi, After upgrading to ceph 14.2.8 I started getting the "1 pool(s) have non-power-of-two pg_num" warning. The offending pool had 100 pgs. I tried to adjust it to 128 but it didn't work: # ceph osd pool set default.rgw.offending pgp_num 128 Error EINVAL: specified pgp_num 128 > pg_num 100 It only worked downwards: # ceph osd pool set default.rgw.offending pgp_num 64 set pool 7 pgp_num to 64 Is that expected? And having reduced my pg_num to 64 (isn't it to small?) should I be prepared for problems in the future? Regards, Rodrigo Severo

2 2

Aborted multipart uploads still visible
by Maks Kowalik 06 Mar '20

06 Mar '20

Hello, I have a problem with one particular bucket that aborting incomplete multipart uploads does not remove them from the list i.e. the storage space is freed up, but 's3cmd multipart' is still showing them. When i retry aborting, I'm getting 404 which is correct. The issue here is that I can only display first 1000 of such uploads, thus can't remove next ones. Does anyone know how can I make the aborted ones disappear from the listing (this was checked on a different bucket and they disappear as expected)? Ceph version is 14.2.4. Kind regards, Maks Kowalik

2 1

Re: How can I fix "object unfound" error?
by Chad William Seys 06 Mar '20

06 Mar '20

> Maybe.... I've marked the object as "lost" and removed the failed > OSD. > > The cluster now is healthy, but I'd like to understand if it's likely > to bother me again in the future. Yeah, I don't know. Within the last month there are 4 separate instances of people mentioning "unfound" object in their cluster. I'm deferring as long as possible any OSD drive upgrades. I ran into the problem when "draining" an OSD. "draining" means remove OSD from crush map, wait for all PG to be stored elsewhere, then replace drive with larger one. Under those circumstances there should be no PG unfound. BTW, are you using cache tiering ? The bug report mentions this, but some people did not have this enabled. Chad.

3 4

Can't add a ceph-mon to existing large cluster
by Dan van der Ster 05 Mar '20

05 Mar '20

Hi all, There's something broken in our env when we try to add new mons to existing clusters, confirmed on two clusters running mimic and nautilus. It's basically this issue https://tracker.ceph.com/issues/42830 In case something is wrong with our puppet manifests, I'm trying to doing it manually. First we --mkfs the mon and start it, but as soon as the new mon starts synchronizing, the existing leader becomes unresponsive and an election is triggered. Here's exactly what I'm doing: # cd /var/lib/ceph/tmp/ # scp cephmon1:/var/lib/ceph/tmp/keyring.mon.cephmon1 keyring.mon.cephmon4 # ceph mon getmap -o monmap # ceph-mon --mkfs -i cephmon4 --monmap monmap --keyrin keyring.mon.cephmon4 --setuser ceph --setgroup ceph # vi /etc/ceph/ceph.conf <add the new mon to ceph.conf like this> [mon.cephmon4] host = cephmon4 mon addr = a.b.c.d:6790 # systemctl start ceph-mon@cephmon4 The log file on the new mon shows it start synchronizing, then immediately the CPU usage on the leader goes to 100% and elections start happening, and ceph health shows mon slow ops. perf top of the ceph-mon with 100% CPU is shown below [1]. On a small nautilus cluster, the new mon gets added withing a minute or so (but not cleanly -- the leader is unresponsive for quite awhile until the new mon joins). debug_mon=20 on the leader doesn't show anything very interesting. On our large mimic cluster we tried waiting more than 10 minutes -- suffering through several mon elections and 100% usage bouncing around between leaders -- until we gave up. I'm pulling my hair out a bit on this -- it's really weird! Did anyone add a new mon to an existing large cluster recently, and it went smoothly? Cheers, Dan [1] 15.12% ceph-mon [.] MonitorDBStore::Transaction::encode 8.95% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::append 8.68% libceph-common.so.0 [.] ceph::buffer::v14_2_0::list::append 7.69% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::release 5.86% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::ptr

4 12

rbd-mirror - which direction?
by Ml Ml 05 Mar '20

05 Mar '20

Hello, i am running luminous and i would like to back up my cluster from Site-A to Site-B (one way) So i decided to mirror it to an off-site ceph cluster. I read: https://docs.ceph.com/docs/luminous/rbd/rbd-mirroring/ But i liked I https://github.com/MiracleMa/Blog/issues/2 a little better. But which parameter/part show the direction it will mirror? Thanks, Mario

1 0

Unexpected recovering after nautilus 14.2.7 -> 14.2.8
by Rainer Krienke 05 Mar '20

05 Mar '20

Hello, at the moment my ceph is still working but in a degraded state after I upgraded one (in 9) hosts from 14.2.7 to 14.2.8 and rebooting this host (node2, one monitor in 3) after the upgrade. Usually before rebooting I set ceph osd set noout ceph osd set nobackfill ceph osd set norecover before rebooting, but I fogot this time. After having realized my error I thought, ok I forgot to set the flags but I configured mon_osd_down_out_interval to 900sec: # ceph config get mon.mon_osd_down_out_interval WHO MASK LEVEL OPTION VALUE RO mon advanced mon_osd_down_out_interval 900 The reboot took 5min so I expected nothing to happen. But it did and now I do not understand why and if there are more timeout values I could/should set to avoid this happening again if I ever should again forget to set the noout , nobackfill, norecover flags prior to a reboot? Thanks if anyone can explain to me what might have happened.... Rainer The current ceph state is: # ceph -s cluster: id: xyz health: HEALTH_WARN Degraded data redundancy: 191629/76527549 objects degraded (0.250%), 18 pgs degraded, 18 pgs undersized services: mon: 3 daemons, quorum node2,node5,node8 (age 51m) mgr: node5(active, since 53m), standbys: node8, node-admin, node2 mds: mycephfs:1 {0=node3=up:active} 2 up:standby osd: 144 osds: 144 up (since 51m), 144 in (since 3M); 48 remapped pgs data: pools: 13 pools, 3460 pgs objects: 12.76M objects, 48 TiB usage: 95 TiB used, 429 TiB / 524 TiB avail pgs: 191629/76527549 objects degraded (0.250%) 3098164/76527549 objects misplaced (4.048%) 3412 active+clean 30 active+remapped+backfill_wait 13 active+undersized+degraded+remapped+backfill_wait 5 active+undersized+degraded+remapped+backfilling io: client: 33 MiB/s rd, 7.2 MiB/s wr, 91 op/s rd, 186 op/s wr recovery: 83 MiB/s, 20 objects/s -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html

3 9

PGs unknown after pool creation (Nautilus 14.2.4/6)
by dg＠terralink.de 05 Mar '20

05 Mar '20

Hello, I have a small ceph cluster running with 3 MON/MGR and 3 OSD hosts. There are also 3 virtual hosts in the crushmap to have a seperate SSD pool. Currently two pools are running, one of that exclusive to the SSD device class. My problem now is, that any new pool I try to create won't become functional as all new pgs are in unknown state. I've tried varying pg number, crush ruleset, size and so on, but nothing helped. my OSDs regularly show an error message like "var/log/ceph/ceph-osd.42.log:2020-03-04 17:09:04.641 7f76240c9700 0 --1- [v2:192.168.44.110:6834/23888,v1:192.168.44.110:6835/23888] >> v1:192.168.44.111:6826/484449 conn(0x55901c805800 0x55901d5bf000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=398 cs=186 l=0).handle_connect_reply_2 connect got BADAUTHORIZER" ,but I don't find any reasons for that (clocks are synchronized). Also one of my mons is two minor versions newer then the other nodes, but would not really like to update the whole cluster right now as I've had somewhat bad experience with the last update :) Does anyone have any idea what I could try ?

1 0

consistency of import-diff
by Stefan Priebe - Profihost AG 05 Mar '20

05 Mar '20

Hello, does anybody know whether there is any mechanism to make sure an image looks like the original after an import-diff? While doing ceph backups on another ceph cluster i currently do a fresh import every 7 days. So i'm sure if something went wrong with import-diff i have a fresh one every 7 days. Otherwise i waste a lot of backup storage. So i wanted to know if there is any way to be sure that the image is OK and save and match the orignal snapshot afterwards. Greets, Stefan

5 10

Re: High memory ceph mgr 14.2.7
by hoannv46＠gmail.com 05 Mar '20

05 Mar '20

I disabled some module in mgr : influx, dashboard, prometheus. After i restart mgr, ram of mgr increase to 20GB in some seconds.

1 0

MDS getting stuck on 'resolve' and 'rejoin'
by Anastasia Belyaeva 04 Mar '20

04 Mar '20

Hello! Our CephFS mds cluster consists of 3 ranks. We had a minor issue with the network the ceph runs on. And after that cephfs became unavaialble: rank 1 and 2 stuck in rejoin rank 0 can't get pass 'resolve' state and keeps getting blacklisted I checked the logs (with debug_mds 5/5) on the rank 0 mds server and found out the following: goes through 'replay' fine then 'resolve' starts and the log gets flooded with messages like -18> 2020-03-04 16:59:56.934 7f77445f0700 5 mds.0.log _submit_thread 442443596224~41 : EImportFinish 0x30000412462 failed -17> 2020-03-04 16:59:56.950 7f77445f0700 5 mds.0.log _submit_thread 442443596285~41 : EImportFinish 0x3000041246c failed -16> 2020-03-04 16:59:56.966 7f77445f0700 5 mds.0.log _submit_thread 442443596346~41 : EImportFinish 0x3000041247b failed -15> 2020-03-04 16:59:56.983 7f77445f0700 5 mds.0.log _submit_thread 442443596407~41 : EImportFinish 0x30000412485 failed then messages about heartbeat errors start showing up (in between) -3210> 2020-03-04 16:59:04.079 7f77485f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -3209> 2020-03-04 16:59:04.079 7f77485f8700 0 mds.beacon.ceph-server11.ibnet Skipping beacon heartbeat to monitors (last acked 8.00204s ago); MDS internal heartbeat is not healthy! the 'flood' ends with messages -14> 2020-03-04 16:59:57.001 7f77455f2700 -1 mds.0.journaler.mdlog(rw) _finish_write_head got (108) Cannot send after transport endpoint shutdown -13> 2020-03-04 16:59:57.001 7f77455f2700 -1 mds.0.journaler.mdlog(rw) handle_write_error (108) Cannot send after transport endpoint shutdown after wich the mds-server gets blacklisted and becomes a 'standby'. And then the same scenario happens to the standby. Also in my attempt to recover the fs I managed to make it worse. I executed cephfs-table-tool 0 reset session and now the mds daemon crashes at 'replay' with the following error -2> 2020-03-04 22:00:54.228 7f4ca6e44700 -1 log_channel(cluster) log [ERR] : error replaying open sessions(1) sessionmap v 7348424 table 0 -1> 2020-03-04 22:00:54.229 7f4ca6e44700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUILD/ceph-14.2.4/src/mds/SessionMap.cc: In function 'void SessionMap::replay_open_sessions(version_t, std::map<client_t, entity_inst_t>&, std::map<client_t, client_metadata_t>&)' thread 7f4ca6e44700 time 2020-03-04 22:00:54.229427 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.4/rpm/el7/BUILD/ceph-14.2.4/src/mds/SessionMap.cc: 750: FAILED ceph_assert(g_conf()->mds_wipe_sessions) ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f4cb7913ac2] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4cb7913c90] 3: (()+0x3b618b) [0x55b43860518b] 4: (EImportStart::replay(MDSRank*)+0x4a8) [0x55b4386805f8] 5: (MDLog::_replay_thread()+0x8ee) [0x55b43861b3ae] 6: (MDLog::ReplayThread::entry()+0xd) [0x55b43838fecd] 7: (()+0x7e25) [0x7f4cb57efe25] 8: (clone()+0x6d) [0x7f4cb46b234d] 0> 2020-03-04 22:00:54.230 7f4ca6e44700 -1 *** Caught signal (Aborted) ** in thread 7f4ca6e44700 thread_name:md_log_replay About the setup ceph version 14.2.4 OS Centos 7.4 3.10.0-693.5.2.el7.x86_64 Any help would be greatly appreciated! Best regards, Anastasia Belyaeva С уважением, Анастасия Беляева Best regards, Anastasia Belyaeva С уважением, Анастасия Беляева

1 0

Need clarification on CephFS, EC Pools, and File Layouts
by Dave Hall 04 Mar '20

04 Mar '20

Hello, This is for a cluster currently running at 14.2.7. Since our cluster is still relatively small we feel a strong need to run our CephFS on an EC Pool (8 + 2) and Crush Failure Domain = OSD to maximize capacity. I have read and re-read https://docs.ceph.com/docs/nautilus/cephfs/createfs/#creating-pools and https://docs.ceph.com/docs/nautilus/cephfs/file-layouts/#file-layouts, but it still isn't quite clear to me. Since this topic is mentioned in the release notes for 14.2.8 I thought I should probably ask so I can configure this correctly. If using a small replicated pool as the default data pool, how does one use a file layout induce the bulk of the data to be stored in the secondary EC data pool? From the links referenced I infer that a file layout is required. Is it possible to have a file layout based solely on size? BTW, we want to do this in a way that we don't have to think about which directory goes with which file size or anything like that. This needs to be an internal detail that is completely hidden from the client. Also, is it possible to insert a replicated data pool as the default on an already deployed CephFS, or will I need to create a new FS and copy the data over? Thanks. -Dave -- Dave Hall Binghamton University

3 3