November 2019 - ceph-users

EC PGs stuck activating, 2^31-1 as OSD ID, automatic recovery not kicking in

by Zoltan Arnold Nagy

Hi, We have a cluster where we mix HDDs and NVMe drives using device classes with a specific crush role for each class. One of our NVMe drives physically died which caused some of our PGs to go into this state: pg 26.ac is stuck undersized for 60830.991784, current state activating+undersized+degraded+remapped, last acting [353,373,368,377,2147483647,350] pg 26.d1 is stuck undersized for 60830.587711, current state activating+undersized+degraded+remapped, last acting [343,2147483647,347,358,366,355] pg 26.e1 is stuck undersized for 60830.980585, current state activating+undersized+degraded+remapped, last acting [340,349,370,2147483647,360,376] ... and so on. Recovery never happened and we had to manually restart all affected OSDs for all PGs stuck in such a state. The 2^31-1 in there seems to indicate an overflow somewhere - the way we were able to figure out where exactly is to query the PG and compare the "up" and "acting" sets - only _one_ of them had the 2^31-1 number in place of the correct OSD number. We restarted that and the PG started doing its job and recovered. The issue seems to be going back to 2015: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-May/001661.html however no solution... I'm more concerned about the cluster not being able to recover (it's a 4+2 EC pool across 12 hosts - plenty of room to heal) than about the weird print-out. The VMs who wanted to access data in any of the affected PGs of course died. Are we missing some settings to let the cluster self-heal even for EC pools? First EC pool in production :) Cheers, Zoltan

4 years

4
4
0 0

EC pool used space high

by Erdem Agaoglu

Hi, I have a rook-provisioned cluster to be used for RBDs only. I have 2 pools named replicated-metadata-pool and ec-data-pool. EC parameters are 6+3. I've been writing some data to this cluster for some time and noticed that the reported usage is not what I was expecting. # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 TOTAL 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL replicated-metadata-pool 1 90 KiB 408 38 MiB 0 1.2 PiB ec-data-pool 2 722 TiB 191.64M 1.2 PiB 25.04 2.4 PiB Since these numbers are rounded a bit too much, I generally use prometheus metrics on mgr, which are as follows: ceph_pool_stored : 793,746 G for ec-data-pool and 92323 for replicated-metadata-pool ceph_pool_stored_raw: 1,190,865 G for ec-data-pool and 99213 for replicated-metadata-pool ceph_cluster_total_used_bytes: 1,329,374 G ceph_cluster_total_used_raw_bytes: 1,333,013 G sum(ceph_bluefs_db_used_bytes) : 3,638 G So ceph_pool_stored for the EC pool is a bit higher than the total used space of the formatted RBDs. I think that's because of the sparse nature and deleted blocks not being fstrimmed yet. That's OK. ceph_pool_stored_raw is almost exactly 1.5x ceph_pool_stored which is what I'd expect considering EC parameters of 6+3. What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case? We also have a ceph_cluster_total_used_raw_bytes metric, I believe to be close to data+metadata. Which is why I tried to show with sum(ceph_bluefs_db_used_bytes). Is that correct? Best, -- erdem agaoglu

4 years

3
5
0 0

[radosgw-admin] Unable to Unlink Bucket From UID

by Mac Wynkoop

Hi all, I seem to be running into an issue when attempting to unlink a bucket from a user; this is my output: user@server ~ $ radosgw-admin bucket unlink --bucket=user_5493/LF-Store --uid=user_5493 failure: 2019-11-26 15:19:48.689 7fda1c2009c0 0 bucket entry point user mismatch, can't unlink bucket: user_5493$BRTC != user_5493 (22) Invalid argument user@server ~ $ I did some searching around, and no one seems to have seen this before. Any ideas? Thanks, Mac

4 years

1
0
0 0

rbd lvm xfs fstrim vs rbd xfs fstrim

by Marc Roos

If I do an fstrim /mount/fs and this is an xsf directly on a rbd device. I can see space being freed instantly with eg. rbd du. However when there is an lvm in between, it looks like this is not freed. I already enabled issue_discards = 1 in lvm.conf but as the comment says, probably only in case of lvremove. Is it possible to get fstrim working with lvm?

4 years

1
1
0 0

ceph user list respone

by Frank R

Hi, For your response: "You should use not more 1Gb for WAL and 30Gb for RocksDB. Numbers ! 3,30,300 (Gb) for block.db is useless. " Do you mean the block.db size should be 3, 30 or 300GB and nothing else? If so, thy not? Thanks, Frank

4 years

2
1
0 0

ceph-fuse non-privileged user mount

by yi zhang

Hi, Recently, I'm trying to mount cephfs using non-privileged users via ceph-fuse，but it‘s always fail. I looked at the code and found that there will be a remount operation when using ceph-fuse mount. Remount will execute the 'mount -i -o remount {mountpoint}' command and that causes the mount to fail. I want to ask How to use non-privileged users to mount cephfs via ceph-fuse? Thanks.

4 years

1
0
0 0

Single mount X multiple mounts

by Rodrigo Severo - Fábrica

Hi, Just starting to use CephFS. I would like to know the impacts of having one single CephFS mount X having several. If I have several subdirectories in my CephFS that should be accessible to different users, with users needing access to different sets of mounts, would it be important for me to try to predefine these sets so to minimize the number of mounts each user needs or can I treat each subdirectory as an independent entity and just mount all necessary for each user and, so, increasing the number of different mounts for each user? Regards, Rodrigo Severo

4 years

2
1
0 0

ceph cache pool question

by Shawn A Kwang

I have a question about ceph cache pools as documented on this page: https://docs.ceph.com/docs/nautilus/dev/cache-pool/ Is the cache pool feature still considered a good idea? Reading some of the email archives I find some discussion of how this caching is not recommended anymore, for version=nautilus. Is that correct? What is my use-case? We are using cephfs, and have a large cephfs pool of HDD with some NVMe drives for bluestore WAL and DB. Total storage is 2.1 PB. Replication=3. There is also a separate metadata pool, as required for cephfs. We have a spare storage server with about 14 TB of NVME drives. Would it be worthwhile to setup an NVMe cache pool for the main cephfs pool? Sincerely, Shawn Kwang -- Associate Scientist Center for Gravitation, Cosmology, and Astrophysics University of Wisconsin-Milwaukee office: +1 414 229 4960 kwangs(a)uwm.edu

4 years

1
0
0 0

FUSE X kernel mounts

by Rodrigo Severo - Fábrica

Hi, I'm just deploying a CephFS service. I would like to know the expected differences between a FUSE and a kernel mount. Why the 2 options? When should I use one and when should I use the other? Regards, Rodrigo Severo

4 years

3
3
0 0

Ceph manager not starting

by Romain Raynaud

Hello please can you help me with that. I'm on ubuntu 18.04 bionic trying to install ceph version 12.2.12 with ceph-deploy to 3 nodes on odroid xu4 (armhf). Creating osd with lvm volumes seem's to work but after starting manager by the command "ceph-deploy mgr create node1", mgr go to starting mode then stay in a health warn state due to no mgr demons active. regards --$ ceph -s --------------------------------------------------------------------- cluster: id: 84f324c2-ac27-4f72-bcb0-7ff1355ee97e health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum node1,node3,node2 mgr: no daemons active osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0B usage: 0B used, 0B / 0B avail pgs: ---/var/log/ceph/ceph-mgr.node1.log------------------------------------------------------------- 2019-11-14 11:08:45.794236 b6f87230 0 ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable), process ceph-mgr, pid 7879 2019-11-14 11:08:45.799472 b6f87230 0 pidfile_write: ignore empty --pid-file 2019-11-14 11:08:45.839575 b6f87230 1 mgr send_beacon standby 2019-11-14 11:08:45.866590 b06c9c30 -1 *** Caught signal (Segmentation fault) ** in thread b06c9c30 thread_name:ms_dispatch ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x302eac) [0x7d6eac] 2: (()+0x25750) [0xb6862750] 3: (_ULarm_step()+0x5b) [0xb67eecec] 4: (()+0x255e8) [0xb6cb05e8] 5: (GetStackTrace(void**, int, int)+0x25) [0xb6cb0a3e] 6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ca536a] 7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ca55e6] 8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ca45ce] 9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x1b) [0xb6ca4760] 10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d) [0xb6ca47de] 11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, unsigned int)+0x51) [0xb6ca6a56] 12: (malloc()+0x22d) [0xb6cb1a8e] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -108> 2019-11-14 11:08:45.776013 b6f87230 5 asok(0x5625320) register_command perfcounters_dump hook 0x55bc090 -107> 2019-11-14 11:08:45.776070 b6f87230 5 asok(0x5625320) register_command 1 hook 0x55bc090 -106> 2019-11-14 11:08:45.776096 b6f87230 5 asok(0x5625320) register_command perf dump hook 0x55bc090 -105> 2019-11-14 11:08:45.776134 b6f87230 5 asok(0x5625320) register_command perfcounters_schema hook 0x55bc090 -104> 2019-11-14 11:08:45.776155 b6f87230 5 asok(0x5625320) register_command perf histogram dump hook 0x55bc090 -103> 2019-11-14 11:08:45.776194 b6f87230 5 asok(0x5625320) register_command 2 hook 0x55bc090 -102> 2019-11-14 11:08:45.776244 b6f87230 5 asok(0x5625320) register_command perf schema hook 0x55bc090 -101> 2019-11-14 11:08:45.776267 b6f87230 5 asok(0x5625320) register_command perf histogram schema hook 0x55bc090 -100> 2019-11-14 11:08:45.776304 b6f87230 5 asok(0x5625320) register_command perf reset hook 0x55bc090 -99> 2019-11-14 11:08:45.776325 b6f87230 5 asok(0x5625320) register_command config show hook 0x55bc090 -98> 2019-11-14 11:08:45.776358 b6f87230 5 asok(0x5625320) register_command config help hook 0x55bc090 -97> 2019-11-14 11:08:45.776380 b6f87230 5 asok(0x5625320) register_command config set hook 0x55bc090 -96> 2019-11-14 11:08:45.776416 b6f87230 5 asok(0x5625320) register_command config get hook 0x55bc090 -95> 2019-11-14 11:08:45.776457 b6f87230 5 asok(0x5625320) register_command config diff hook 0x55bc090 -94> 2019-11-14 11:08:45.776479 b6f87230 5 asok(0x5625320) register_command config diff get hook 0x55bc090 -93> 2019-11-14 11:08:45.776513 b6f87230 5 asok(0x5625320) register_command log flush hook 0x55bc090 -92> 2019-11-14 11:08:45.776534 b6f87230 5 asok(0x5625320) register_command log dump hook 0x55bc090 -91> 2019-11-14 11:08:45.776568 b6f87230 5 asok(0x5625320) register_command log reopen hook 0x55bc090 -90> 2019-11-14 11:08:45.776679 b6f87230 5 asok(0x5625320) register_command dump_mempools hook 0x5757b04 -89> 2019-11-14 11:08:45.794236 b6f87230 0 ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable), process ceph-mgr, pid 7879 -88> 2019-11-14 11:08:45.799472 b6f87230 0 pidfile_write: ignore empty --pid-file -87> 2019-11-14 11:08:45.800376 b6f87230 1 finished global_init_daemonize -86> 2019-11-14 11:08:45.813287 b6f87230 5 asok(0x5625320) init /var/run/ceph/ceph-mgr.node1.asok -85> 2019-11-14 11:08:45.813359 b6f87230 5 asok(0x5625320) bind_and_listen /var/run/ceph/ceph-mgr.node1.asok -84> 2019-11-14 11:08:45.813867 b6f87230 5 asok(0x5625320) register_command 0 hook 0x55bc180 -83> 2019-11-14 11:08:45.813920 b6f87230 5 asok(0x5625320) register_command version hook 0x55bc180 -82> 2019-11-14 11:08:45.813954 b6f87230 5 asok(0x5625320) register_command git_version hook 0x55bc180 -81> 2019-11-14 11:08:45.813994 b6f87230 5 asok(0x5625320) register_command help hook 0x55bc178 -80> 2019-11-14 11:08:45.814085 b6f87230 5 asok(0x5625320) register_command get_command_descriptions hook 0x55bc170 -79> 2019-11-14 11:08:45.814371 b3ed0c30 5 asok(0x5625320) entry start -78> 2019-11-14 11:08:45.819066 b36cfc30 2 Event(0x55be068 nevent=5000 time_id=1).set_owner idx=0 owner=3010264112 -77> 2019-11-14 11:08:45.819281 b2ecec30 2 Event(0x55be488 nevent=5000 time_id=1).set_owner idx=1 owner=3001871408 -76> 2019-11-14 11:08:45.819484 b26cdc30 2 Event(0x55be1c8 nevent=5000 time_id=1).set_owner idx=2 owner=2993478704 -75> 2019-11-14 11:08:45.821210 b6f87230 1 Processor -- start -74> 2019-11-14 11:08:45.821472 b6f87230 1 -- - start start -73> 2019-11-14 11:08:45.821506 b6f87230 10 monclient: build_initial_monmap -72> 2019-11-14 11:08:45.821667 b6f87230 10 monclient: init -71> 2019-11-14 11:08:45.821796 b6f87230 5 adding auth protocol: cephx -70> 2019-11-14 11:08:45.821826 b6f87230 10 monclient: auth_supported 2 method cephx -69> 2019-11-14 11:08:45.822606 b6f87230 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mgr/ceph-node1/keyring -68> 2019-11-14 11:08:45.822947 b6f87230 10 monclient: _reopen_session rank -1 -67> 2019-11-14 11:08:45.823131 b6f87230 10 monclient(hunting): picked mon.noname-b con 0x5804d00 addr [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -66> 2019-11-14 11:08:45.823297 b6f87230 10 monclient(hunting): picked mon.noname-c con 0x5805a00 addr [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -65> 2019-11-14 11:08:45.823437 b6f87230 1 -- - --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5600680 con 0 -64> 2019-11-14 11:08:45.823555 b6f87230 1 -- - --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5600820 con 0 -63> 2019-11-14 11:08:45.823646 b6f87230 10 monclient(hunting): _renew_subs -62> 2019-11-14 11:08:45.825773 b26cdc30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 learned_addr learned my addr [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 -61> 2019-11-14 11:08:45.826773 b26cdc30 2 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -60> 2019-11-14 11:08:45.827140 b2ecec30 2 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_acked_seq 0 vs out_seq 0 -59> 2019-11-14 11:08:45.828839 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 1 0x5630900 mon_map magic: 0 v1 -58> 2019-11-14 11:08:45.829033 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 1 ==== mon_map magic: 0 v1 ==== 469+0+0 (726272991 0 0) 0x5630900 con 0x5805a00 -57> 2019-11-14 11:08:45.829141 b06c9c30 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -56> 2019-11-14 11:08:45.829085 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 2 0x560e540 auth_reply(proto 2 0 (0) Success) v1 -55> 2019-11-14 11:08:45.829165 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 1 0x5630a80 mon_map magic: 0 v1 -54> 2019-11-14 11:08:45.829225 b06c9c30 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1 -53> 2019-11-14 11:08:45.829244 b06c9c30 10 monclient(hunting): dump: epoch 1 fsid 84f324c2-ac27-4f72-bcb0-7ff1355ee97e last_changed 2019-11-13 14:18:49.133327 created 2019-11-13 14:18:49.133327 0: [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:6789/0 mon.node1 1: [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 mon.fpgh3 2: [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 mon.fpgh2 -52> 2019-11-14 11:08:45.829331 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 2 0x560e700 auth_reply(proto 2 0 (0) Success) v1 -51> 2019-11-14 11:08:45.829402 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3710817385 0 0) 0x560e540 con 0x5805a00 -50> 2019-11-14 11:08:45.829512 b06c9c30 10 monclient(hunting): my global_id is 24280 -49> 2019-11-14 11:08:45.829791 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5600d00 con 0 -48> 2019-11-14 11:08:45.829910 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.2 [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 1 ==== mon_map magic: 0 v1 ==== 469+0+0 (726272991 0 0) 0x5630a80 con 0x5804d00 -47> 2019-11-14 11:08:45.829982 b06c9c30 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -46> 2019-11-14 11:08:45.830052 b06c9c30 10 monclient(hunting): got monmap 1, mon.fpgh2 is now rank 2 -45> 2019-11-14 11:08:45.830085 b06c9c30 10 monclient(hunting): dump: epoch 1 fsid 84f324c2-ac27-4f72-bcb0-7ff1355ee97e last_changed 2019-11-13 14:18:49.133327 created 2019-11-13 14:18:49.133327 0: [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:6789/0 mon.node1 1: [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 mon.fpgh3 2: [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 mon.fpgh2 -44> 2019-11-14 11:08:45.830244 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.2 [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2346605455 0 0) 0x560e700 con 0x5804d00 -43> 2019-11-14 11:08:45.830343 b06c9c30 10 monclient(hunting): my global_id is 24308 -42> 2019-11-14 11:08:45.830535 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x5600680 con 0 -41> 2019-11-14 11:08:45.832185 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 3 0x560e8c0 auth_reply(proto 2 0 (0) Success) v1 -40> 2019-11-14 11:08:45.832327 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3339254479 0 0) 0x560e8c0 con 0x5805a00 -39> 2019-11-14 11:08:45.832834 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5600d00 con 0 -38> 2019-11-14 11:08:45.833078 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 3 0x560e700 auth_reply(proto 2 0 (0) Success) v1 -37> 2019-11-14 11:08:45.833228 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.2 [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (3081092587 0 0) 0x560e700 con 0x5804d00 -36> 2019-11-14 11:08:45.833816 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5600820 con 0 -35> 2019-11-14 11:08:45.836366 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 4 0x560ea80 auth_reply(proto 2 0 (0) Success) v1 -34> 2019-11-14 11:08:45.836509 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 751+0+0 (1301292739 0 0) 0x560ea80 con 0x5805a00 -33> 2019-11-14 11:08:45.837249 b2ecec30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=87 cs=1 l=1). rx mon.2 seq 4 0x560e8c0 auth_reply(proto 2 0 (0) Success) v1 -32> 2019-11-14 11:08:45.837359 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN pgs=87 cs=1 l=1).mark_down -31> 2019-11-14 11:08:45.837425 b06c9c30 2 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86b6]:6789/0 conn(0x5804d00 :-1 s=STATE_OPEN pgs=87 cs=1 l=1)._stop -30> 2019-11-14 11:08:45.837558 b06c9c30 1 monclient: found mon.fpgh3 -29> 2019-11-14 11:08:45.837623 b06c9c30 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -28> 2019-11-14 11:08:45.837669 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- mon_subscribe({mgrmap=0+,monmap=0+}) v2 -- 0x55beb00 con 0 -27> 2019-11-14 11:08:45.837797 b06c9c30 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-11-14 11:08:15.837792) -26> 2019-11-14 11:08:45.837860 b06c9c30 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -25> 2019-11-14 11:08:45.837923 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x5600680 con 0 -24> 2019-11-14 11:08:45.838036 b6f87230 5 monclient: authenticate success, global_id 24280 -23> 2019-11-14 11:08:45.838182 b6f87230 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -22> 2019-11-14 11:08:45.838290 b6f87230 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -21> 2019-11-14 11:08:45.838619 b6f87230 5 asok(0x5625320) register_command objecter_requests hook 0x55bc1d8 -20> 2019-11-14 11:08:45.838736 b6f87230 10 monclient: _renew_subs -19> 2019-11-14 11:08:45.838770 b6f87230 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -18> 2019-11-14 11:08:45.838816 b6f87230 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- mon_subscribe({osdmap=0}) v2 -- 0x55bec60 con 0 -17> 2019-11-14 11:08:45.839247 b6f87230 5 asok(0x5625320) register_command mds_requests hook 0xbe83e770 -16> 2019-11-14 11:08:45.839310 b6f87230 5 asok(0x5625320) register_command mds_sessions hook 0xbe83e770 -15> 2019-11-14 11:08:45.839347 b6f87230 5 asok(0x5625320) register_command dump_cache hook 0xbe83e770 -14> 2019-11-14 11:08:45.839383 b6f87230 5 asok(0x5625320) register_command kick_stale_sessions hook 0xbe83e770 -13> 2019-11-14 11:08:45.839420 b6f87230 5 asok(0x5625320) register_command status hook 0xbe83e770 -12> 2019-11-14 11:08:45.839575 b6f87230 1 mgr send_beacon standby -11> 2019-11-14 11:08:45.839646 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 5 0x55fde00 mgrmap(e 11813) v1 -10> 2019-11-14 11:08:45.839756 b06c9c30 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 <== mon.1 [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 5 ==== mgrmap(e 11813) v1 ==== 237+0+0 (2937343514 0 0) 0x55fde00 con 0x5805a00 -9> 2019-11-14 11:08:45.839778 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 6 0x5630900 mon_map magic: 0 v1 -8> 2019-11-14 11:08:45.840302 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 7 0x560e700 auth_reply(proto 2 0 (0) Success) v1 -7> 2019-11-14 11:08:45.840645 b26cdc30 5 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 >> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 conn(0x5805a00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=67 cs=1 l=1). rx mon.1 seq 8 0x5600680 osd_map(45..45 src has 1..45) v3 -6> 2019-11-14 11:08:45.840927 b6f87230 10 monclient: _send_mon_message to mon.fpgh3 at [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -5> 2019-11-14 11:08:45.840980 b6f87230 1 -- [2a01:e0a:c:b9f0:21e:6ff:fe30:c8fc]:0/1860748422 --> [2a01:e0a:c:b9f0:21e:6ff:fe36:86ad]:6789/0 -- mgrbeacon mgr.node1(84f324c2-ac27-4f72-bcb0-7ff1355ee97e,24280, -, 0) v6 -- 0x55d6400 con 0 -4> 2019-11-14 11:08:45.841111 b6f87230 4 mgr init Complete. -3> 2019-11-14 11:08:45.841220 b06c9c30 4 mgr ms_dispatch standby mgrmap(e 11813) v1 -2> 2019-11-14 11:08:45.841266 b06c9c30 4 mgr handle_mgr_map received map epoch 11813 -1> 2019-11-14 11:08:45.841282 b06c9c30 4 mgr handle_mgr_map active in map: 0 active is 0 0> 2019-11-14 11:08:45.866590 b06c9c30 -1 *** Caught signal (Segmentation fault) ** in thread b06c9c30 thread_name:ms_dispatch ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0x302eac) [0x7d6eac] 2: (()+0x25750) [0xb6862750] 3: (_ULarm_step()+0x5b) [0xb67eecec] 4: (()+0x255e8) [0xb6cb05e8] 5: (GetStackTrace(void**, int, int)+0x25) [0xb6cb0a3e] 6: (tcmalloc::PageHeap::GrowHeap(unsigned int)+0xb9) [0xb6ca536a] 7: (tcmalloc::PageHeap::New(unsigned int)+0x79) [0xb6ca55e6] 8: (tcmalloc::CentralFreeList::Populate()+0x71) [0xb6ca45ce] 9: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x1b) [0xb6ca4760] 10: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x6d) [0xb6ca47de] 11: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, unsigned int)+0x51) [0xb6ca6a56] 12: (malloc()+0x22d) [0xb6cb1a8e] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mgr.node1.log --- end dump of recent events --- *Romain Raynaud*

4 years

1
0
0 0

2023

2022

2021

2020

2019

ceph-users November 2019