February 2020 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

bluestore_default_buffered_write = true
by Adam Koczarski 20 Jul '20

20 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

ERROR: osd init failed: (1) Operation not permitted
by Ml Ml 22 Jun '20

22 Jun '20

Hello List, first of all: Yes - i made mistakes. Now i am trying to recover :-/ I had a healthy 3 node cluster which i wanted to convert to a single one. My goal was to reinstall a fresh 3 Node cluster and start with 2 nodes. I was able to healthy turn it from a 3 Node Cluster to a 2 Node cluster. Then the problems began. I started to change size=1 and min_size=1. Health was okay until here. Then over sudden both nodes got fenced...one node refused to boot, mons where missing, etc...to make long story short, here is where i am right now: root@node03:~ # ceph -s cluster b3be313f-d0ef-42d5-80c8-6b41380a47e3 health HEALTH_WARN 53 pgs stale 53 pgs stuck stale monmap e4: 2 mons at {0=10.15.15.3:6789/0,1=10.15.15.2:6789/0} election epoch 298, quorum 0,1 1,0 osdmap e6097: 14 osds: 9 up, 9 in pgmap v93644673: 512 pgs, 1 pools, 1193 GB data, 304 kobjects 1088 GB used, 32277 GB / 33366 GB avail 459 active+clean 53 stale+active+clean root@node03:~ # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 32.56990 root default -2 25.35992 host node03 0 3.57999 osd.0 up 1.00000 1.00000 5 3.62999 osd.5 up 1.00000 1.00000 6 3.62999 osd.6 up 1.00000 1.00000 7 3.62999 osd.7 up 1.00000 1.00000 8 3.62999 osd.8 up 1.00000 1.00000 19 3.62999 osd.19 up 1.00000 1.00000 20 3.62999 osd.20 up 1.00000 1.00000 -3 7.20998 host node02 3 3.62999 osd.3 up 1.00000 1.00000 4 3.57999 osd.4 up 1.00000 1.00000 1 0 osd.1 down 0 1.00000 9 0 osd.9 down 0 1.00000 10 0 osd.10 down 0 1.00000 17 0 osd.17 down 0 1.00000 18 0 osd.18 down 0 1.00000 my main mistakes seemd to be: -------------------------------- ceph osd out osd.1 ceph auth del osd.1 systemctl stop ceph-osd@1 ceph osd rm 1 umount /var/lib/ceph/osd/ceph-1 ceph osd crush remove osd.1 As far as i can tell, ceph waits and needs data from that OSD.1 (which i removed) root@node03:~ # ceph health detail HEALTH_WARN 53 pgs stale; 53 pgs stuck stale pg 0.1a6 is stuck stale for 5086.552795, current state stale+active+clean, last acting [1] pg 0.142 is stuck stale for 5086.552784, current state stale+active+clean, last acting [1] pg 0.1e is stuck stale for 5086.552820, current state stale+active+clean, last acting [1] pg 0.e0 is stuck stale for 5086.552855, current state stale+active+clean, last acting [1] pg 0.1d is stuck stale for 5086.552822, current state stale+active+clean, last acting [1] pg 0.13c is stuck stale for 5086.552791, current state stale+active+clean, last acting [1] [...] SNIP [...] pg 0.e9 is stuck stale for 5086.552955, current state stale+active+clean, last acting [1] pg 0.87 is stuck stale for 5086.552939, current state stale+active+clean, last acting [1] When i try to start ODS.1 manually, i get: -------------------------------------------- 2020-02-10 18:48:26.107444 7f9ce31dd880 0 ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 10210 2020-02-10 18:48:26.134417 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) backend xfs (magic 0x58465342) 2020-02-10 18:48:26.184202 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is supported and appears to work 2020-02-10 18:48:26.184209 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2020-02-10 18:48:26.184526 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2020-02-10 18:48:26.184585 7f9ce31dd880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_feature: extsize is disabled by conf 2020-02-10 18:48:26.309755 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2020-02-10 18:48:26.633926 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.642185 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.664273 7f9ce31dd880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2020-02-10 18:48:26.732154 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for clients 2020-02-10 18:48:26.732163 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400 was 8705, adjusting msgr requires for mons 2020-02-10 18:48:26.732167 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for osds 2020-02-10 18:48:26.732179 7f9ce31dd880 0 osd.1 6002 load_pgs 2020-02-10 18:48:31.939810 7f9ce31dd880 0 osd.1 6002 load_pgs opened 53 pgs 2020-02-10 18:48:31.940546 7f9ce31dd880 -1 osd.1 6002 log_to_monitors {default=true} 2020-02-10 18:48:31.942471 7f9ce31dd880 1 journal close /var/lib/ceph/osd/ceph-1/journal 2020-02-10 18:48:31.969205 7f9ce31dd880 -1 ESC[0;31m ** ERROR: osd init failed: (1) Operation not permittedESC[0m Its mounted: /dev/sdg1 3.7T 127G 3.6T 4% /var/lib/ceph/osd/ceph-1 Is there any way i can get the OSD.1 back in? Thanks a lot, mario

2 1

Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
by Dan van der Ster 10 Jun '20

10 Jun '20

This means it has been applied: # ceph osd dump -f json | jq .require_osd_release "nautilus" -- dan On Mon, Feb 17, 2020 at 11:10 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > How do you check if you issued this command in the past? > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Excessive write load on mons after upgrade > from 12.2.13 -> 14.2.7 > > Hi Peter, > > could be a totally different problem but did you run the command "ceph > osd require-osd-release nautilus" after the upgrade? > We had poor performance after upgrading to nautilus and running this > command fixed it. The same was reported by others for previous updates. > Here is my original message regarding this issue: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPV… > > We did not observe the master election problem though. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

Help! ceph-mon is blocked after shutting down and ip address changed
by occj＠qq.com 04 Jun '20

04 Jun '20

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) os :CentOS Linux release 7.7.1908 (Core) single node ceph cluster with 1 mon,1mgr,1 mds,1rgw and 12osds , but only cephfs is used. ceph -s is blocked after shutting down the machine (192.168.0.104), then ip address changed to 192.168.1.6 I created the monmap with monmap tool and update the ceph.conf , hosts file and then start ceph-mon. and the ceph-mon log: ... 2019-12-11 08:57:45.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1285.14s 2019-12-11 08:57:50.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1290.14s 2019-12-11 08:57:55.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1295.14s 2019-12-11 08:58:00.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1300.14s 2019-12-11 08:58:05.172 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1305.14s 2019-12-11 08:58:10.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1310.14s 2019-12-11 08:58:15.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1315.14s 2019-12-11 08:58:20.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1320.14s 2019-12-11 08:58:25.174 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1325.14s ... I changed IP back to 192.168.0.104 yeasterday, but all the same. # cat /etc/ceph/ceph.conf [client.libvirt] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor log file = /var/log/ceph/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor [client.rgw.ceph-node1.rgw0] host = ceph-node1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-node1.rgw0.log rgw frontends = beast endpoint=192.168.1.6:8080 rgw thread pool size = 512 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.1.0/24 fsid = e384e8e6-94d5-4812-bfbb-d1b0468bdef5 mon host = [v2:192.168.1.6:3300,v1:192.168.1.6:6789] mon initial members = ceph-node1 osd crush chooseleaf type = 0 osd pool default crush rule = -1 public network = 192.168.1.0/24 [osd] osd memory target = 7870655146

2 1

Radosgw PubSub Traffic
by Dustin Guerrero 02 Jun '20

02 Jun '20

Hey all, We’ve been running some benchmarks against Ceph which we deployed using the Rook operator in Kubernetes. Everything seemed to scale linearly until a point where I see a single OSD receiving much higher CPU load than the other OSDs (nearly 100% saturation). After some investigation we noticed a ton of pubsub traffic in the strace coming from the RGW pods like so: [pid 22561] sendmsg(77, {msg_name(0)=NULL, msg_iov(3)=[{"\21\2)\0\0\0\10\0:\1\0\0\10\0\0\0\0\0\10\0\0\0\0\0\0\20\0\0-\321\211K"..., 73}, {"\200\0\0\0pubsub.user.ceph-user-wwITOk"..., 314}, {"\0\303\34[\360\314\233\2138\377\377\377\377\377\377\377\377", 17}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL|MSG_MORE <unfinished …> I’ve checked other OSDs and only a single OSD receives these messages. I suspect its creating a bottleneck. Does anyone have an idea on why these are being generated or how to stop them? The pubsub sync module doesn’t appear to be enabled, and our benchmark is doing simple gets/puts/deletes. We’re running Ceph 14.2.5 nautilus Thank you!

2 2

OSD backups and recovery
by Ludek Navratil 30 May '20

30 May '20

HI all, what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local filesystem and backup the content using Amanda. I believe that there must a better way to do this but I couldn't find it in docs. I know that one option is to setup an archive zone, but it requires an additional ceph cluster that needs to be maintained and looked after. I would rather avoid that. How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me to recover the data correctly? Many thanks,Ludek

4 5

General question CephFS or RBD
by Willi Schiegel 29 May '20

29 May '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

2 1

Ceph and iSCSI
by Bobby 29 May '20

29 May '20

Hi all, I am new to Ceph. But I have a some good understanding of iSCSI protocol. I will dive into Ceph because it looks promising. I am particularly interested in Ceph-RBD. I have a request. Can you please tell me, if any, what are the common similarities between iSCSI and Ceph. If someone has to work on a common model for iSCSI and Ceph, what would be those significant points you would suggest to someone who has some understanding of iSCSI? Looking forward to answers. Thanks in advance :-) BR

2 1

Very bad performance on a ceph rbd pool via iSCSI to VMware esx
by Salsa 29 May '20

29 May '20

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 2

Ceph and Windows - experiences or suggestions
by Lars Täuber 29 May '20

29 May '20

Hi there! I got the task to connect a Windows client to our existing ceph cluster. I'm looking for experiences or suggestions from the community. There came two possibilities to my mind: 1. iSCSI Target on RBD exported to Windows 2. NFS-Ganesha on CephFS exported to Windows Is there a third way exporting a ceph cluster to a windows machine? I have some experiences with CephFS. We have a small cluster successfully running for linux clients. I don't have experiences with RBD or iSCSI. The Windows machine will use the space for backups. The kind of data is unknown. I expect the data to be a MS-SQL dump and user data from a sharepoint system. The Windows admin does not care whether NFS or iSCSI is used. I'd be happy if some of you could share experiences. Thanks Lars

10 10

ceph with rdma can not mount with kernel
by 李亚锋 29 May '20

29 May '20

2 1

Ceph on CentOS 8?
by Jan Kasprzak 29 May '20

29 May '20

Hello, Ceph users, does anybody use Ceph on recently released CentOS 8? Apparently there are no el8 packages neither at download.ceph.com, nor in the native CentOS package tree. I am thinking about upgrading my cluster to C8 (because of other software running on it apart from Ceph). Do el7 packages simply work? Can they be rebuilt using rpmbuild --rebuild? Or is running Ceph on C8 more complicated than that? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | sir_clive> I hope you don't mind if I steal some of your ideas? laryross> As far as stealing... we call it sharing here. --from rcgroups

9 8

rbd image naming convention
by Palanisamy 29 May '20

29 May '20

Hello Team, We've integrated Ceph cluster storage with Kubernetes and provisioning volumes through rbd-provisioner. When we're creating volumes from yaml files in Kubernetes, pv > pvc > mounting to pod, In kubernetes end pvc are showing as meaningful naming convention as per yaml file defined. But in ceph cluster, rbd image name is creating with dynamic uid. During troubleshooting time, this will be tedious to find exact rbd image. Please find the provisioning logs in below pasted snippet. kubectl get pods,pv,pvc NAME READY STATUS RESTARTS AGE pod/sleepypod 1/1 Running 0 4m9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO Delete Bound default/test-dyn-pvc ceph-rbd 4m9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-dyn-pvc Bound pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO ceph-rbd 4m11s *rbd-provisioner logs* I1121 10:59:15.009012 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2" I1121 10:59:15.009092 1 controller.go:1087] provision "default/test-dyn-pvc" class "ceph-rbd": volume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" provisioned I1121 10:59:15.009138 1 controller.go:1101] provision "default/test-dyn-pvc" class "ceph-rbd": trying to save persistentvvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" I1121 10:59:15.020418 1 controller.go:1108] provision "default/test-dyn-pvc" class "ceph-rbd": persistentvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" saved I1121 10:59:15.020476 1 controller.go:1149] provision "default/test-dyn-pvc" class "ceph-rbd": succeeded I1121 10:59:15.020802 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-dyn-pvc", UID:"cd37d2d6-cecc-4a05-9736-c8d80abde7f5", APIVersion:"v1", ResourceVersion:"24545639", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 *rbd image details in Ceph cluster end* rbd -p kube ls --long NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2 1 GiB 2 is there way to setup proper naming convention for rbd image as well during kubernetes deployment itself. Kubernetes version: v1.15.5 Ceph cluster version: 14.2.2 nautilus (stable) *Best Regards,* *Palanisamy*

2 2

cephfs file layouts, empty objects in first data pool
by Håkan T Johansson 28 May '20

28 May '20

Hi, running 14.2.6, debian buster (backports). Have set up a cephfs with 3 data pools and one metadata pool: myfs_data, myfs_data_hdd, myfs_data_ssd, and myfs_metadata. The data of all files are with the use of ceph.dir.layout.pool either stored in the pools myfs_data_hdd or myfs_data_ssd. This has also been checked by dumping the ceph.file.layout.pool attributes of all files. The filesystem has 1617949 files and 36042 directories. There are however approximately as many objects in the first pool created for the cephfs, myfs_data, as there are files. They also becomes more or fewer as files are created or deleted (so cannot be some leftover from earlier exercises). Note how the USED size is reported as 0 bytes, correctly reflecting that no file data is stored in them. POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR myfs_data 0 B 1618229 0 4854687 0 0 0 2263590 129 GiB 23312479 124 GiB 0 B 0 B myfs_data_hdd 831 GiB 136309 0 408927 0 0 0 106046 200 GiB 269084 277 GiB 0 B 0 B myfs_data_ssd 43 GiB 1552412 0 4657236 0 0 0 181468 2.3 GiB 4661935 12 GiB 0 B 0 B myfs_metadata 1.2 GiB 36096 0 108288 0 0 0 4828623 82 GiB 1355102 143 GiB 0 B 0 B Is this expected? I was assuming that in this scenario, all objects, both their data and any keys would be either in the metadata pool, or the two pools where the objects are stored. Is it some additional metadata keys that are stored in the first created data pool for cephfs? This would not be so nice in case the osd selection rules for it are using worse disks than the data itself... Btw: is there any tool to see the amount of key value data size associated with a pool? 'ceph osd df' gives omap and meta for osds, but not broken down per pool. Best regards, Håkan

4 6

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

Re: MDS: obscene buffer_anon memory use when scanning lots of files
by John Madden 15 Apr '20

15 Apr '20

Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested: ~$ ceph tell mds.mds1 heap stats 2020-02-10 16:52:44.313 7fbda2cae700 0 client.59208005 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:44.337 7fbda3cb0700 0 client.59249562 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 tcmalloc heap stats:------------------------------------------------ MALLOC: 50000388656 (47684.1 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 174879528 ( 166.8 MiB) Bytes in central cache freelist MALLOC: + 14511680 ( 13.8 MiB) Bytes in transfer cache freelist MALLOC: + 14089320 ( 13.4 MiB) Bytes in thread cache freelists MALLOC: + 90534048 ( 86.3 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 50294403232 (47964.5 MiB) Actual memory used (physical + swap) MALLOC: + 50987008 ( 48.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 50345390240 (48013.1 MiB) Virtual address space used MALLOC: MALLOC: 260018 Spans in use MALLOC: 20 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ~$ ceph tell mds.mds1 heap release 2020-02-10 16:52:47.205 7f037eff5700 0 client.59249625 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:47.237 7f037fff7700 0 client.59249634 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 releasing free RAM back to system. The pools over 15 minutes or so: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2045, "bytes": 3069493686 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2445, "bytes": 3111162538 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 7850, "bytes": 7658678767 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 12274, "bytes": 11436728978 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 13747, "bytes": 11539478519 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 14615, "bytes": 13859676992 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 23267, "bytes": 22290063830 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 44944, "bytes": 40726959425 } And one about a minute after the heap release showing continued growth: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 50694, "bytes": 47343942094 } This is on a single active MDS with 2 standbys, scan for about a million files with about 20 parallel threads on two clients, open and read each if it exists. On Wed, Jan 22, 2020 at 8:25 AM John Madden <jmadden.com(a)gmail.com> wrote: > > > Couldn't John confirm that this is the issue by checking the heap stats and triggering the release via > > > > ceph tell mds.mds1 heap stats > > ceph tell mds.mds1 heap release > > > > (this would be much less disruptive than restarting the MDS) > > That was my first thought as well, but `release` doesn't appear to do > anything in this case. > > John

3 8

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

Identify slow ops
by Thomas Schneider 27 Mar '20

27 Mar '20

Hi, the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing. root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d) data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops There's no error on services (mgr, mon, osd). Can you please advise how to identify the root cause of this slow ops? THX

3 6

upmap balancer
by Frank R 16 Mar '20

16 Mar '20

Hi all, Does using the Upmap balancer require that all OSDs be the same size (per device class)? thx Frank

4 7

Re: Kworker 100% with ceph-msgr (after upgrade to 14.2.6?)
by Ilya Dryomov 15 Mar '20

15 Mar '20

On Tue, Jan 14, 2020 at 10:31 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I think this is new since I upgraded to 14.2.6. kworker/7:3 100% > > [@~]# echo l > /proc/sysrq-trigger > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192e7e>] > [<ffffffffb0192e7e>] generic_swap+0x1e/0x30 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a20 EFLAGS: 00000206 > [Tue Jan 14 10:05:08 2020] RAX: 0000000000000072 RBX: 0000000000000060 > RCX: 0000000000000072 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000006 RSI: ffffa0c788914a7a > RDI: ffffa0c788914a42 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a20 R08: 0000000000000028 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe94984224500 > R12: ffffa0c788914a40 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0c788914a18 > R15: ffffa0c788914a78 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > > > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192200>] > [<ffffffffb0192200>] __x86_indirect_thunk_rax+0x0/0x20 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a28 EFLAGS: 00000286 > [Tue Jan 14 10:05:08 2020] RAX: ffffffffb0192e60 RBX: 0000000000000010 > RCX: 0000000000000010 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000008 RSI: ffffa0d0e108b828 > RDI: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a98 R08: 0000000000000000 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe949a9842280 > R12: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0d0e108b818 > R15: ffffa0d0e108b828 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] ? sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 Hi Marc, It's busy processing snapshot contexts, I think it's more or less the same issue that you reported in "ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)". Did "ceph: build_snap_context 100020c9287 ffff911a9a26bd00 fail -12" errors go away? How many snapshots do you have now? Did you file a tracker ticket? It's better to have all details in one place rather than scattered across different threads on the list. Adding Jeff and Zheng, who can take a closer look. Thanks, Ilya

2 2

centos7 / nautilus where to get kernel 5.5 from?
by Marc Roos 12 Mar '20

12 Mar '20

I have default centos7 setup with nautilus. I have been asked to install 5.5 to check a 'bug'. Where should I get this from? I read that the elrepo kernel is not compiled like rhel.

2 2

RGWReshardLock::lock failed to acquire lock ret=-16
by Josh Haft 12 Mar '20

12 Mar '20

Hi, Currently running Mimic 13.2.5. We had reports this morning of timeouts and failures with PUT and GET requests to our Ceph RGW cluster. I found these messages in the RGW log: RGWReshardLock::lock failed to acquire lock on bucket_name:bucket_instance ret=-16 NOTICE: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is still resharding, please retry Which were preceded by many of these, which I think are normal/expected. check_bucket_shards: resharding needed: stats.num_objects=6415879 shard max_objects=6400000 Our RGW cluster sits behind haproxy which notified me approx 90 seconds after the first 'resharding needed' message that no backends were available. It appears this dynamic reshard process caused the RGWs to lock up for a period of time. Roughly 2 minutes later the reshard error messages stop and operation returns to normal. Looking back through previous RGW logs, I see a similar event from about a week ago, on the same bucket. We have several buckets with shard counts exceeding 1k (this one only has 128), and much larger object counts, so clearly this isn't the first time dynamic sharding has been invoked on this cluster. Has anyone seen this? I expect it will come up again, and can turn up debugging if that'll help. Thanks for any assistance! Josh

1 1

Re: Nautilus OSD memory consumption?
by Nigel Williams 11 Mar '20

11 Mar '20

On Thu, 27 Feb 2020 at 06:27, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > If the heap stats reported by telling the OSD `heap stats` is large, telling each `heap release` may be useful. I suspect a TCMALLOC shortcoming. osd.158 tcmalloc heap stats:------------------------------------------------ MALLOC: 5722761448 ( 5457.7 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 311621760 ( 297.2 MiB) Bytes in central cache freelist MALLOC: + 26242992 ( 25.0 MiB) Bytes in transfer cache freelist MALLOC: + 62721768 ( 59.8 MiB) Bytes in thread cache freelists MALLOC: + 113340608 ( 108.1 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 6236688576 ( 5947.8 MiB) Actual memory used (physical + swap) MALLOC: + 21415870464 (20423.8 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 27652559040 (26371.5 MiB) Virtual address space used MALLOC: MALLOC: 394518 Spans in use MALLOC: 37 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory.

2 2

High CPU usage by ceph-mgr in 14.2.6
by jbardgett＠godaddy.com 07 Mar '20

07 Mar '20

After upgrading one of our clusters from Luminous 12.2.12 to Nautilus 14.2.6, I am seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). The way we found this was due to Prometheus being unable to report out certain pieces of data, specifically OSD Usage, OSD Apply and Commit Latency. Which are all similar issues people were having in previous versions of Nautilus. Bryan Stillwell reported this previously on a separate cluster, 14.2.5, we have here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VW3GNVJGOOW… That issue was resolved with the upgrade to 14.2.6. We are seeing a similar issue on this other cluster with a couple differences. This cluster has 1900+ OSD in it, the previous one had 300+ The top user is libceph-common, instead of mmap 4.86% libceph-common.so.0 [.] EventCenter::create_time_event 2.78% [kernel] [k] nmi 2.64% libstdc++.so.6.0.19 [.] __dynamic_cast On all our other clusters that have been upgraded to 14.2.6 we are not experiencing this issue, the next largest being 800+ OSD. We feel this is related to the size of the cluster, similarly to the previous report. Anyone else experiencing this and/or can provide some direction on how to go about resolving this? Thanks, Joe

6 7

Re: Problems with ragosgw
by Eric Ivancich 29 Feb '20

29 Feb '20

Hi Mohamed, > On Jan 22, 2020, at 10:05 AM, mohamed zayan <mohamed.zayan19(a)gmail.com <mailto:mohamed.zayan19@gmail.com>> wrote: > > Currently I have a cluster of 2 nodes on two raspberrypi 3 devices. > pi1 is admin/mon/mgr/osd > > pi2 is osd > > I am currently trying to run radosgw on pi2. I have failed multiple times > > /var/lib/ceph/radosgw# /usr/bin/radosgw -f --cluster ceph --name > client.rgw.pi2 --setuser ceph --setgroup ceph > Thread::try_create(): pthread_create failed with error > 11/tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc <http://thread.cc/>: > In function 'void Thread::create(const char *, size_t)' thread > 71114000 time 2020-01-22 14:58:13.793803 > /tmp/release/Raspbian/WORKDIR/ceph-12.2.9-38-gaeeb23362d/src/common/Thread.cc <http://thread.cc/>: > 152: FAILED assert(ret == 0) I’m not too familiar with the Raspberry Pi platform or Raspbian. But there are some clues in the output you provided, specifically this text — “ Thread::try_create(): pthread_create failed with error 11”. I assume these low valued error codes are consistent between Fedora and Raspbian, and so the code 11 means: #define EAGAIN 11 /* Try again */ I found the following: https://stackoverflow.com/questions/47078106/pthread-create-fails-with-eaga… <https://stackoverflow.com/questions/47078106/pthread-create-fails-with-eaga…> So that seems to leave 3 possibilities: 1. RGW is asking for more resources that your platform is able to provide. a. There are limits as to the number of threads. b. There may be a memory limit, since each thread needs to maintain its own stack. 2. There is a bug in RGW where completed threads are not thread_join’d by their parents. a. This seems unlikely as this appears to happen during start-up before threads are likely done with their work. 3. Florian Weimer identifies a kernel bug. I suspect it’s #1, so you might want to try reducing the number of threads rgw uses by lowering the value of rgw_thread_pool_size in your configuration. Eric -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, USA

1 0

recover ceph-mon
by xsempresu＠gmail.com 29 Feb '20

29 Feb '20

I have test one-node ceph cluster with 4 osds under assumption to add the second node just before production. Linux 4.19.0-6-amd64 - debian 10 - ceph version 12.2.11 Unfortunately, system drive was broken before it. I recovered the system from full backup. Since no changes was performed to the cluster configuration after that backup, I hoped that it works. For the reasons I can't understand first few seconds after boot ceph status was OK (134 active+clean, 2 active+clean+scrubbing+deep), but a minute later status changed to: # ceph status cluster: id: e02f2885-946b-46c8-91d5-146dd724ecaf health: HEALTH_WARN 1 filesystem is degraded 2 osds down 1 slice (2 osds) down Reduced data availability: 136 pgs inactive, 15 pgs peering services: mon: 1 daemons, quorum rbd0 mgr: rbd0(active) mds: fs-1/1/1 up {0=rbd0=up:replay} osd: 5 osds: 1 up, 3 in data: pools: 2 pools, 136 pgs objects: 118.53k objects, 429GiB usage: 7.15TiB used, 3.77TiB / 10.9TiB avail pgs: 88.971% pgs unknown 11.029% pgs not active 121 unknown 15 peering # ceph osd dump epoch 1983 fsid e02f2885-946b-46c8-91d5-146dd724ecaf created 2019-08-16 15:14:07.783009 modified 2020-02-29 13:55:39.212461 flags sortbitwise,recovery_deletes,purged_snapdirs crush_version 27 full_ratio 0.97 backfillfull_ratio 0.94 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client jewel require_osd_release luminous pool 1 'fs_data' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 1595 flags hashpspool stripe_width 0 application cephfs pool 2 'fs_meta' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1595 flags hashpspool stripe_width 0 application cephfs max_osd 5 osd.0 down out weight 0 up_from 1970 up_thru 1973 down_at 1975 last_clean_interval [1949,1963) 192.168.101.111:6806/440 192.168.101.111:6807/440 192.168.101.111:6808/440 192.168.101.111:6809/440 autoout,exists 78eaeb63-47c9-4962-b8ff-46607921f4f6 osd.1 down in weight 1 up_from 1970 up_thru 1970 down_at 1975 last_clean_interval [1952,1963) 192.168.101.111:6801/439 192.168.101.111:6810/439 192.168.101.111:6811/439 192.168.101.111:6812/439 exists c4c4c85d-f537-4199-823b-b7ab01c78f03 osd.2 down in weight 1 up_from 1969 up_thru 1975 down_at 1976 last_clean_interval [1946,1963) 192.168.101.111:6802/441 192.168.101.111:6803/441 192.168.101.111:6804/441 192.168.101.111:6805/441 exists bd66a9c3-bfa4-4352-816e-2e4cd86389f3 osd.3 down out weight 0 up_from 1617 up_thru 1619 down_at 1631 last_clean_interval [1602,1610) 192.168.101.111:6805/933 192.168.101.111:6806/933 192.168.101.111:6807/933 192.168.101.111:6808/933 exists f247115b-c6d5-49b1-9b0e-e799c50be379 osd.4 up in weight 1 up_from 1973 up_thru 1973 down_at 1972 last_clean_interval [1956,1963) 192.168.101.111:6813/442 192.168.101.111:6814/442 192.168.101.111:6815/442 192.168.101.111:6816/442 exists,up c208221e-1228-4247-a742-0c16ce01d38f blacklist 192.168.101.111:6800/2636437603 expires 2020-03-01 13:26:01.809132 "ceph pg query" of any PG didn't response. I can't find any errors in journalctl or in /var/log/ceph/* I wonder why only osd 4 up, what means outoout, why 15 pgs are peering, where to search detail information, is it a way to restore data. Please help me to understand what happend and how to restore data if it possible.

1 0

Question about ceph-balancer and OSD reweights
by shubjero 28 Feb '20

28 Feb '20

Hi all, I'm running a Ceph Mimic cluster 13.2.6 and we use the ceph-balancer in upmap mode. This cluster is fairly old and pre-Mimic we used to set osd reweights to balance the standard deviation of the cluster. Since moving to Mimic about 9 months ago I enabled the ceph-balancer with upmap mode and let it do its thing but I did not think about setting the previously modified reweights back to 1.00000 (not sure if this is fine or would have been a best practice?) Does the ceph-balancer in upmap mode manage the osd reweight dynamically? Just wondering if I need to proactively go back and set all non 1.00000 reweights to 1.00000. Thanks all, I hope that makes sense!

3 4

Stately MDS Transitions
by DHilsbos＠performair.com 28 Feb '20

28 Feb '20

All; We just started really fiddling with CephFS on our production cluster (Nautilus - 14.2.5 / 14.2.6), and I have a question... Is there a command / set of commands that transitions a standby-replay MDS server to the active role, while swapping the active MDS to standby-replay, or even just standby? I'm looking for a way to seamlessly, and without down time, prepare the active MDS to go offline (reboot) as part of planned /periodic maintenance. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 3

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Lincoln Bryant 28 Feb '20

28 Feb '20

Hi all, I just want to report that I was able to recover from the incomplete PG after exporting the PGs and using the dangerous osd_find_best_info_ignore_history_les option on the appropriate OSDs :) Things seem okay for now, fingers crossed. When marking hit set archive objects lost, I was using 'mark_unfound_lost' command with the 'revert' flag rather than 'delete'. I wonder if the objects were reverted to previous versions, which were causing my OSDs to assert? Here's that error again for reference: 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 flags = none tried to add 1365222'2867906(1363674'2866712) flags = delete Anyhow I'd like to +1 the issue Paul reported here: https://tracker.ceph.com/issues/44286 . This seems totally in line with what we are seeing as well. --Lincoln ________________________________ From: Lincoln Bryant <lincolnb(a)uchicago.edu> Sent: Thursday, February 27, 2020 12:05 PM To: Sage Weil <sage(a)newdream.net>; Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 It seems that one of the down PGs was able to recover just fine, but the other OSD went into "incomplete" state after export-and-removing the affected PG from the down OSD. I've still got the exported data from the pg, although re-importing it to the OSD again causes the crashes. What's the best way to proceed here? It would be very very preferable not to lose the data. Thanks, Lincoln ________________________________ From: Lincoln Bryant <lincolnb(a)uchicago.edu> Sent: Thursday, February 27, 2020 9:26 AM To: Sage Weil <sage(a)newdream.net>; Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 Thanks Sage, I can try that. Admittedly I'm not sure how to tell if these two PG can recover without this particular OSD. Note, it seems like there is still an underlying related issue, with hit set archives popping up as unfound objects on my cluster as in Paul's ticket. In total I had about 10-20 such objects in my cluster, most of which were ok with marking them as unfound/lost. I've had this happen once before a few months ago with unfound hit set archives but was able to safely mark the object lost and move on (or so I thought..?). --Lincoln ________________________________ From: Sage Weil <sage(a)newdream.net> Sent: Thursday, February 27, 2020 9:01 AM To: Paul Emmerich <paul.emmerich(a)croit.io> Cc: Lincoln Bryant <lincolnb(a)uchicago.edu>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 If the pg in question can recover without that OSD, I would use use ceph-objectstore-tool to export and remove it, and then move on. I hit a similar issue on my system (due to a bunch in an early octopus build) and it was super tedious to fix up manually (needed patched code and manual modifications of the store). sage On Thu, 27 Feb 2020, Paul Emmerich wrote: > Crash happens in PG::activate, so it's unrelated to IO etc. > > My first approach here would be to read the code and try to understand > why it crashes/what the exact condition is that is violated here. > It looks like something that can probably be fixed by fiddling around > with ceph-objectstore-tool (but you should try to understand what > exactly is happening before running random ceph-objectstore-tool > commands) > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io<http://www.croit.io> > Tel: +49 89 1896585 90 > > On Thu, Feb 27, 2020 at 1:15 PM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > Thanks Paul. > > > > I was able to mark many of the unfound ones as lost, but I'm still stuck with one unfound and OSD assert at this point. > > > > I've tried setting many of the OSD options to pause all cluster I/O, backfilling, rebalancing, tiering agent, etc to try to avoid hitting the assert but alas this one OSD is still crashing. The OSD in question does manage to log quite a bit of things before crashing. > > > > Is there any way for me to delete this or create a dummy object in RADOS that will let this OSD come up, I wonder? > > > > --Lincoln > > > > OBJECT_UNFOUND 1/793053192 objects unfound (0.000%) > > pg 36.1755 has 1 unfound objects > > PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs down > > pg 36.1153 is down+remapped, acting [299] > > pg 36.2047 is down+remapped, acting [242] > > > > -2> 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 f > > lags = none tried to add 1365222'2867906(1363674'2866712) flags = delete > > > > > > ________________________________ > > From: Paul Emmerich <paul.emmerich(a)croit.io> > > Sent: Thursday, February 27, 2020 5:27 AM > > To: Lincoln Bryant <lincolnb(a)uchicago.edu> > > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > > Subject: Re: [ceph-users] Cache tier OSDs crashing due to unfound hitset object 14.2.7 > > > > I've also encountered this issue, but luckily without the crashing > > OSDs, so marking as lost resolved it for us. > > > > See https://tracker.ceph.com/issues/44286 > > > > > > Paul > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io<http://www.croit.io> > > Tel: +49 89 1896585 90 > > > > On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > > > Hello Ceph experts, > > > > > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > > > > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > > > > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > > > > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > > > > > Thanks, > > > Lincoln Bryant > > > > > > > > > [1] > > > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > > > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > > > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > > > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > > > 2: (()+0x4dddd7) [0x55955fdafdd7] > > > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 15: (()+0x7e25) [0x7ff5669bae25] > > > 16: (clone()+0x6d) [0x7ff565a9a34d] > > > > > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > > > in thread 7ff52edaa700 thread_name:tp_osd_tp > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (()+0xf5e0) [0x7ff5669c25e0] > > > 2: (gsignal()+0x37) [0x7ff5659d71f7] > > > 3: (abort()+0x148) [0x7ff5659d88e8] > > > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > > > 5: (()+0x4dddd7) [0x55955fdafdd7] > > > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 18: (()+0x7e25) [0x7ff5669bae25] > > > 19: (clone()+0x6d) [0x7ff565a9a34d] > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > > > --- logging levels --- > > > 0/ 5 none > > > 0/ 1 lockdep > > > 0/ 1 context > > > 1/ 1 crush > > > 1/ 5 mds > > > 1/ 5 mds_balancer > > > 1/ 5 mds_locker > > > 1/ 5 mds_log > > > 1/ 5 mds_log_expire > > > 1/ 5 mds_migrator > > > 0/ 1 buffer > > > 0/ 1 timer > > > 0/ 1 filer > > > 0/ 1 striper > > > 0/ 1 objecter > > > 0/ 5 rados > > > 0/ 5 rbd > > > 0/ 5 rbd_mirror > > > 0/ 5 rbd_replay > > > 0/ 5 journaler > > > 0/ 5 objectcacher > > > 0/ 5 client > > > 1/ 5 osd > > > 0/ 5 optracker > > > 0/ 5 objclass > > > 1/ 3 filestore > > > 1/ 3 journal > > > 0/ 0 ms > > > 1/ 5 mon > > > 0/10 monc > > > 1/ 5 paxos > > > 0/ 5 tp > > > 1/ 5 auth > > > 1/ 5 crypto > > > 1/ 1 finisher > > > 1/ 1 reserver > > > 1/ 5 heartbeatmap > > > 1/ 5 perfcounter > > > 1/ 5 rgw > > > 1/ 5 rgw_sync > > > 1/10 civetweb > > > 1/ 5 javaclient > > > 1/ 5 asok > > > 1/ 1 throttle > > > 0/ 0 refs > > > 1/ 5 xio > > > 1/ 5 compressor > > > 1/ 5 bluestore > > > 1/ 5 bluefs > > > 1/ 3 bdev > > > 1/ 5 kstore > > > 4/ 5 rocksdb > > > 4/ 5 leveldb > > > 4/ 5 memdb > > > 1/ 5 kinetic > > > 1/ 5 fuse > > > 1/ 5 mgr > > > 1/ 5 mgrc > > > 1/ 5 dpdk > > > 1/ 5 eventtrace > > > 1/ 5 prioritycache > > > -2/-2 (syslog threshold) > > > -1/-1 (stderr threshold) > > > max_recent 10000 > > > max_new 1000 > > > log_file /var/log/ceph/ceph-osd.299.log > > > --- end dump of recent events --- > > > > > > > > > [2] > > > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > > > "key": "", > > > "snapid": -2, > > > "hash": 12827, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1352209'2834660", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > > > "key": "", > > > "snapid": -2, > > > "hash": 12874, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361100'2822063", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > > > "key": "", > > > "snapid": -2, > > > "hash": 4316, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361089'2838543", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

continued warnings: Large omap object found
by Seth Galitzer 28 Feb '20

28 Feb '20

I do not have a large ceph cluster, only 4 nodes plus a mon/mgr with 48 OSDs. I have one data pool and one metadata pool with a total of about 140TB of usable storage. I have maybe 30 or so clients. The rest of my systems connect via a host that is a ceph client and then reshares through samba and nfs-ganesha. I'm not using rgw anywhere. I'm running the latest stable release of nautilus (14.2.7) and have had it in production since August 2019. All ceph nodes and the smb/nfs host are running centos7 with latest patches. Other clients are a mix of debian and ubuntu. For the last several weeks, I have been getting the warning "Large omap object found" off and on. I've been resolving it by gradually increasing the value of osd_deep_scrub_large_omap_object_key_threshold and then running a deep scrub on the affected pg. I have now increased this threshold to 1000000 and am wondering if I should keep doing this or if there is another problem that needs to be addressed. The affected pg has been different most times, but they are all on the same osd and with the same mds object. Here's an excerpt from my current set of logs to show what I'm seeing: # zgrep -i "large omap object found" /var/log/ceph/ceph.log* /var/log/ceph/ceph.log:2020-02-27 06:02:01.761641 osd.40 (osd.40) 1578 : cluster [WRN] Large omap object found. Object: 2:654134d2:::mds0_openfiles.0:head PG: 2.4b2c82a6 (2.26) Key count: 1048576 Size (bytes): 46403355 /var/log/ceph/ceph.log:2020-02-27 16:18:00.328869 osd.40 (osd.40) 1585 : cluster [WRN] Large omap object found. Object: 2:654134d2:::mds0_openfiles.0:head PG: 2.4b2c82a6 (2.26) Key count: 1048559 Size (bytes): 46407183 /var/log/ceph/ceph.log-20200227.gz:2020-02-26 19:56:24.972431 osd.40 (osd.40) 1450 : cluster [WRN] Large omap object found. Object: 2:c9647462:::mds0_openfiles.1:head PG: 2.462e2693 (2.13) Key count: 939236 Size (bytes): 40179994 /var/log/ceph/ceph.log-20200227.gz:2020-02-26 21:14:16.497161 osd.40 (osd.40) 1460 : cluster [WRN] Large omap object found. Object: 2:c9647462:::mds0_openfiles.1:head PG: 2.462e2693 (2.13) Key count: 939232 Size (bytes): 40179796 /var/log/ceph/ceph.log-20200227.gz:2020-02-26 21:15:06.399267 osd.40 (osd.40) 1464 : cluster [WRN] Large omap object found. Object: 2:c9647462:::mds0_openfiles.1:head PG: 2.462e2693 (2.13) Key count: 939231 Size (bytes): 40179756 Unfortunately, older logs have already been rotated out, but if memory serves correctly, they had similar messages. As you can see, the key count continues to increase. Last week, I bumped the threshold to 750000 to clear the warning. Before that, I had bumped to 500000. It looks to me like something isn't getting cleaned up like it's supposed to. I haven't been using ceph long enough to figure out what that might be. Do I continue to bump the key threshold and not worry about the warnings, or is there something going on that needs to be corrected? At what point is the threshold too high? If the problem is due to a specific client not closing files, is it possible to identify that client and attempt to reset it? Any advice is welcome. I'm happy to provide additional data if needed. Thanks. Seth -- Seth Galitzer Systems Coordinator Computer Science Department Kansas State University http://www.cs.ksu.edu/~sgsax sgsax(a)ksu.edu 785-532-7790

2 2

Best way to merge crush buckets?
by Adrien Georget 28 Feb '20

28 Feb '20

Hi all, I'm looking for the best way to merge/remap existing host buckets into one. I'm running a Ceph Nautilus cluster used as a Ceph Cinder backend with 2 pools "volume-service" and "volume-recherche" both with dedicated OSDs : |host cccephnd00x-service {|| || id -2 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item osd.0 weight 7.275|| || item osd.6 weight 7.275|| ||}|| ||host cccephnd00x-recherche {|| || id -3 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item osd.11 weight 7.266|| || item osd.17 weight 7.266|| || item osd.22 weight 7.266|| || item osd.27 weight 7.266|| ||} ... ||root service {|| || id -26 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item cccephnd001-service weight 14.550|| || ...|| || item cccephnd006-service weight 14.550|| || ||}|| ||root recherche {|| || id -27 # do not change unnecessarily|| || alg straw2|| || hash 0 # rjenkins1|| || item cccephnd001-recherche weight 29.064|| || ...|| || item cccephnd006-recherche weight 29.064|| ||}|| ||rule HAService {|| || id 1|| || type replicated|| || min_size 1|| || max_size 10|| || step take service|| || step chooseleaf firstn 0 type host|| || step emit|| ||}|| ||rule Recherche {|| || id 2|| || type replicated|| || min_size 1|| || max_size 10|| || step take recherche|| || step chooseleaf firstn 0 type host|| || step emit|| ||} $ ceph osd pool get volumes-service crush_rule crush_rule: HAService $ ceph osd pool get volumes-recherche crush_rule crush_rule: Recherche | The pool "volume-service" used to work with SSD cache tiering but we decided to stop using it. So I would like to keep these 2 pools but merge the buckets /host cccephnd00X-service/ and /host cccephnd00X-recherche/ into one /cccephnd00X-cinder/ for better performance (more OSDs assigned to pools). In theory, migrate to this kind of crushmap: |host cccephnd00x-cinder {|| || id -22 || || alg straw2|| || hash 0 # rjenkins1|| || item osd.0 weight 7.275|| || item osd.6 weight 7.275||| || item osd.11 weight 7.266|| || item osd.17 weight 7.266|| || item osd.22 weight 7.266|| || item osd.27 weight 7.266|| | ||}|| |... ||root cinder {|| || id -23 || || alg straw2|| || hash 0 # rjenkins1|| || item cccephnd001-|||||cinder| weight 29.064|| || ...|| || item cccephnd006-|||||cinder| weight 29.064|| ||}|| ||rule Cinder {|| || id 24|| || type replicated|| || min_size 1|| || max_size 10|| || step take ||||||cinder|||| || step chooseleaf firstn 0 type host|| || step emit|| ||} |||$ ceph osd pool get volumes-service crush_rule crush_rule: ||||||cinder|||| || $ ceph osd pool get volumes-recherche crush_rule crush_rule: ||||||cinder|||| || |In practice, I'm thinking about creating the new host bucket/root/rule and then change the crush_rule used for the 2 pools to the new one. And then delete the old one. Do you think that's can be done easily? (And without losing existing data?) I guess there will be a huge rebalance activity but I don't have many choice. Or do you have any other suggestions? Cheers, Adrien |||| ||| |||

1 0

SSD considerations for block.db and WAL
by Christian Wahl 28 Feb '20

28 Feb '20

Hi everyone, we currently have 6 OSDs with 8TB HDDs split across 3 hosts. The main usage is KVM-Images. To improve speed we planned on putting the block.db and WAL onto NVMe-SSDs. The plan was to put 2x1TB in each host. One option I thought of was to RAID 1 them for better redundancy, I don't know how high the risk is of corrupting the block.db by one failed SSD block. Or should I just one for WAL+block.db and use the other one as fast storage? Thank you all very much! Christian

3 2

rgw lifecycle process is not fast enough
by quexian da 28 Feb '20

28 Feb '20

ceph version: 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) I set up a ceph cluster and I'm uploading objects through rgw with a speed of 60 objects/s. I added some lifecycle rules to buckets so that my disks will not be used up. However, after I set "debug_rgw" to 5 and run `radosgw-admin lc process`, I found from the logs that the deletion is not parallel (only objects in one bucket is deleted in the same time) and speed is much slower than my upload speed (60 objects/s), the disk usage is still growing bigger and bigger. Could you please tell me what should I do to speed up the lifecycle process? Thanks in advance!

1 0

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Lincoln Bryant 27 Feb '20

27 Feb '20

It seems that one of the down PGs was able to recover just fine, but the other OSD went into "incomplete" state after export-and-removing the affected PG from the down OSD. I've still got the exported data from the pg, although re-importing it to the OSD again causes the crashes. What's the best way to proceed here? It would be very very preferable not to lose the data. Thanks, Lincoln ________________________________ From: Lincoln Bryant <lincolnb(a)uchicago.edu> Sent: Thursday, February 27, 2020 9:26 AM To: Sage Weil <sage(a)newdream.net>; Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 Thanks Sage, I can try that. Admittedly I'm not sure how to tell if these two PG can recover without this particular OSD. Note, it seems like there is still an underlying related issue, with hit set archives popping up as unfound objects on my cluster as in Paul's ticket. In total I had about 10-20 such objects in my cluster, most of which were ok with marking them as unfound/lost. I've had this happen once before a few months ago with unfound hit set archives but was able to safely mark the object lost and move on (or so I thought..?). --Lincoln ________________________________ From: Sage Weil <sage(a)newdream.net> Sent: Thursday, February 27, 2020 9:01 AM To: Paul Emmerich <paul.emmerich(a)croit.io> Cc: Lincoln Bryant <lincolnb(a)uchicago.edu>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: Re: [ceph-users] Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7 If the pg in question can recover without that OSD, I would use use ceph-objectstore-tool to export and remove it, and then move on. I hit a similar issue on my system (due to a bunch in an early octopus build) and it was super tedious to fix up manually (needed patched code and manual modifications of the store). sage On Thu, 27 Feb 2020, Paul Emmerich wrote: > Crash happens in PG::activate, so it's unrelated to IO etc. > > My first approach here would be to read the code and try to understand > why it crashes/what the exact condition is that is violated here. > It looks like something that can probably be fixed by fiddling around > with ceph-objectstore-tool (but you should try to understand what > exactly is happening before running random ceph-objectstore-tool > commands) > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io<http://www.croit.io> > Tel: +49 89 1896585 90 > > On Thu, Feb 27, 2020 at 1:15 PM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > Thanks Paul. > > > > I was able to mark many of the unfound ones as lost, but I'm still stuck with one unfound and OSD assert at this point. > > > > I've tried setting many of the OSD options to pause all cluster I/O, backfilling, rebalancing, tiering agent, etc to try to avoid hitting the assert but alas this one OSD is still crashing. The OSD in question does manage to log quite a bit of things before crashing. > > > > Is there any way for me to delete this or create a dummy object in RADOS that will let this OSD come up, I wonder? > > > > --Lincoln > > > > OBJECT_UNFOUND 1/793053192 objects unfound (0.000%) > > pg 36.1755 has 1 unfound objects > > PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs down > > pg 36.1153 is down+remapped, acting [299] > > pg 36.2047 is down+remapped, acting [242] > > > > -2> 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 f > > lags = none tried to add 1365222'2867906(1363674'2866712) flags = delete > > > > > > ________________________________ > > From: Paul Emmerich <paul.emmerich(a)croit.io> > > Sent: Thursday, February 27, 2020 5:27 AM > > To: Lincoln Bryant <lincolnb(a)uchicago.edu> > > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > > Subject: Re: [ceph-users] Cache tier OSDs crashing due to unfound hitset object 14.2.7 > > > > I've also encountered this issue, but luckily without the crashing > > OSDs, so marking as lost resolved it for us. > > > > See https://tracker.ceph.com/issues/44286 > > > > > > Paul > > > > -- > > Paul Emmerich > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > croit GmbH > > Freseniusstr. 31h > > 81247 München > > www.croit.io<http://www.croit.io> > > Tel: +49 89 1896585 90 > > > > On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > > > Hello Ceph experts, > > > > > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > > > > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > > > > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > > > > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > > > > > Thanks, > > > Lincoln Bryant > > > > > > > > > [1] > > > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > > > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > > > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > > > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > > > 2: (()+0x4dddd7) [0x55955fdafdd7] > > > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 15: (()+0x7e25) [0x7ff5669bae25] > > > 16: (clone()+0x6d) [0x7ff565a9a34d] > > > > > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > > > in thread 7ff52edaa700 thread_name:tp_osd_tp > > > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > > 1: (()+0xf5e0) [0x7ff5669c25e0] > > > 2: (gsignal()+0x37) [0x7ff5659d71f7] > > > 3: (abort()+0x148) [0x7ff5659d88e8] > > > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > > > 5: (()+0x4dddd7) [0x55955fdafdd7] > > > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > > 18: (()+0x7e25) [0x7ff5669bae25] > > > 19: (clone()+0x6d) [0x7ff565a9a34d] > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > > > --- logging levels --- > > > 0/ 5 none > > > 0/ 1 lockdep > > > 0/ 1 context > > > 1/ 1 crush > > > 1/ 5 mds > > > 1/ 5 mds_balancer > > > 1/ 5 mds_locker > > > 1/ 5 mds_log > > > 1/ 5 mds_log_expire > > > 1/ 5 mds_migrator > > > 0/ 1 buffer > > > 0/ 1 timer > > > 0/ 1 filer > > > 0/ 1 striper > > > 0/ 1 objecter > > > 0/ 5 rados > > > 0/ 5 rbd > > > 0/ 5 rbd_mirror > > > 0/ 5 rbd_replay > > > 0/ 5 journaler > > > 0/ 5 objectcacher > > > 0/ 5 client > > > 1/ 5 osd > > > 0/ 5 optracker > > > 0/ 5 objclass > > > 1/ 3 filestore > > > 1/ 3 journal > > > 0/ 0 ms > > > 1/ 5 mon > > > 0/10 monc > > > 1/ 5 paxos > > > 0/ 5 tp > > > 1/ 5 auth > > > 1/ 5 crypto > > > 1/ 1 finisher > > > 1/ 1 reserver > > > 1/ 5 heartbeatmap > > > 1/ 5 perfcounter > > > 1/ 5 rgw > > > 1/ 5 rgw_sync > > > 1/10 civetweb > > > 1/ 5 javaclient > > > 1/ 5 asok > > > 1/ 1 throttle > > > 0/ 0 refs > > > 1/ 5 xio > > > 1/ 5 compressor > > > 1/ 5 bluestore > > > 1/ 5 bluefs > > > 1/ 3 bdev > > > 1/ 5 kstore > > > 4/ 5 rocksdb > > > 4/ 5 leveldb > > > 4/ 5 memdb > > > 1/ 5 kinetic > > > 1/ 5 fuse > > > 1/ 5 mgr > > > 1/ 5 mgrc > > > 1/ 5 dpdk > > > 1/ 5 eventtrace > > > 1/ 5 prioritycache > > > -2/-2 (syslog threshold) > > > -1/-1 (stderr threshold) > > > max_recent 10000 > > > max_new 1000 > > > log_file /var/log/ceph/ceph-osd.299.log > > > --- end dump of recent events --- > > > > > > > > > [2] > > > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > > > "key": "", > > > "snapid": -2, > > > "hash": 12827, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1352209'2834660", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > > > "key": "", > > > "snapid": -2, > > > "hash": 12874, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361100'2822063", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > > > { > > > "num_missing": 1, > > > "num_unfound": 1, > > > "objects": [ > > > { > > > "oid": { > > > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > > > "key": "", > > > "snapid": -2, > > > "hash": 4316, > > > "max": 0, > > > "pool": 36, > > > "namespace": ".ceph-internal" > > > }, > > > "need": "1361089'2838543", > > > "have": "0'0", > > > "flags": "none", > > > "locations": [] > > > } > > > ], > > > "more": false > > > } > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

osdmap::decode crc error -- 13.2.7 -- most osds down
by Dan van der Ster 27 Feb '20

27 Feb '20

Hi, My turn. We suddenly have a big outage which is similar/identical to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-August/036519.html Some of the osds are runnable, but most crash when they start -- crc error in osdmap::decode. I'm able to extract an osd map from a good osd and it decodes well with osdmaptool: # ceph-objectstore-tool --op get-osdmap --data-path /var/lib/ceph/osd/ceph-680/ --file osd.680.map But when I try on one of the bad osds I get: # ceph-objectstore-tool --op get-osdmap --data-path /var/lib/ceph/osd/ceph-666/ --file osd.666.map terminate called after throwing an instance of 'ceph::buffer::malformed_input' what(): buffer::malformed_input: bad crc, actual 822724616 != expected 2334082500 *** Caught signal (Aborted) ** in thread 7f600aa42d00 thread_name:ceph-objectstor ceph version 13.2.7 (71bd687b6e8b9424dd5e5974ed542595d8977416) mimic (stable) 1: (()+0xf5f0) [0x7f5ffefc45f0] 2: (gsignal()+0x37) [0x7f5ffdbae337] 3: (abort()+0x148) [0x7f5ffdbafa28] 4: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f5ffe4be7d5] 5: (()+0x5e746) [0x7f5ffe4bc746] 6: (()+0x5e773) [0x7f5ffe4bc773] 7: (()+0x5e993) [0x7f5ffe4bc993] 8: (OSDMap::decode(ceph::buffer::list::iterator&)+0x160e) [0x7f6000f4168e] 9: (OSDMap::decode(ceph::buffer::list&)+0x31) [0x7f6000f42e31] 10: (get_osdmap(ObjectStore*, unsigned int, OSDMap&, ceph::buffer::list&)+0x1d0) [0x55d30a489190] 11: (main()+0x5340) [0x55d30a3aae70] 12: (__libc_start_main()+0xf5) [0x7f5ffdb9a505] 13: (()+0x3a0f40) [0x55d30a483f40] Aborted (core dumped) I think I want to inject the osdmap, but can't: # ceph-objectstore-tool --op set-osdmap --data-path /var/lib/ceph/osd/ceph-666/ --file osd.680.map osdmap (#-1:b65b78ab:::osdmap.2983572:0#) does not exist. How do I do this? Thanks for any help! dan

2 6

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Paul Emmerich 27 Feb '20

27 Feb '20

Crash happens in PG::activate, so it's unrelated to IO etc. My first approach here would be to read the code and try to understand why it crashes/what the exact condition is that is violated here. It looks like something that can probably be fixed by fiddling around with ceph-objectstore-tool (but you should try to understand what exactly is happening before running random ceph-objectstore-tool commands) Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Feb 27, 2020 at 1:15 PM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > Thanks Paul. > > I was able to mark many of the unfound ones as lost, but I'm still stuck with one unfound and OSD assert at this point. > > I've tried setting many of the OSD options to pause all cluster I/O, backfilling, rebalancing, tiering agent, etc to try to avoid hitting the assert but alas this one OSD is still crashing. The OSD in question does manage to log quite a bit of things before crashing. > > Is there any way for me to delete this or create a dummy object in RADOS that will let this OSD come up, I wonder? > > --Lincoln > > OBJECT_UNFOUND 1/793053192 objects unfound (0.000%) > pg 36.1755 has 1 unfound objects > PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs down > pg 36.1153 is down+remapped, acting [299] > pg 36.2047 is down+remapped, acting [242] > > -2> 2020-02-27 06:13:12.265 7f0824f1c700 0 0x55ed866481e0 36.2047 unexpected need for 36:e2040000:.ceph-internal::hit_set_36.2047_archive_2020-02-25 19%3a32%3a07.171593_2020-02-25 21%3a27%3a36.268116:head have 1363674'2866712 f > lags = none tried to add 1365222'2867906(1363674'2866712) flags = delete > > > ________________________________ > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Thursday, February 27, 2020 5:27 AM > To: Lincoln Bryant <lincolnb(a)uchicago.edu> > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] Cache tier OSDs crashing due to unfound hitset object 14.2.7 > > I've also encountered this issue, but luckily without the crashing > OSDs, so marking as lost resolved it for us. > > See https://tracker.ceph.com/issues/44286 > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > > > Hello Ceph experts, > > > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > > > Thanks, > > Lincoln Bryant > > > > > > [1] > > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > > 2: (()+0x4dddd7) [0x55955fdafdd7] > > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > 15: (()+0x7e25) [0x7ff5669bae25] > > 16: (clone()+0x6d) [0x7ff565a9a34d] > > > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > > in thread 7ff52edaa700 thread_name:tp_osd_tp > > > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > > 1: (()+0xf5e0) [0x7ff5669c25e0] > > 2: (gsignal()+0x37) [0x7ff5659d71f7] > > 3: (abort()+0x148) [0x7ff5659d88e8] > > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > > 5: (()+0x4dddd7) [0x55955fdafdd7] > > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > > 18: (()+0x7e25) [0x7ff5669bae25] > > 19: (clone()+0x6d) [0x7ff565a9a34d] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > --- logging levels --- > > 0/ 5 none > > 0/ 1 lockdep > > 0/ 1 context > > 1/ 1 crush > > 1/ 5 mds > > 1/ 5 mds_balancer > > 1/ 5 mds_locker > > 1/ 5 mds_log > > 1/ 5 mds_log_expire > > 1/ 5 mds_migrator > > 0/ 1 buffer > > 0/ 1 timer > > 0/ 1 filer > > 0/ 1 striper > > 0/ 1 objecter > > 0/ 5 rados > > 0/ 5 rbd > > 0/ 5 rbd_mirror > > 0/ 5 rbd_replay > > 0/ 5 journaler > > 0/ 5 objectcacher > > 0/ 5 client > > 1/ 5 osd > > 0/ 5 optracker > > 0/ 5 objclass > > 1/ 3 filestore > > 1/ 3 journal > > 0/ 0 ms > > 1/ 5 mon > > 0/10 monc > > 1/ 5 paxos > > 0/ 5 tp > > 1/ 5 auth > > 1/ 5 crypto > > 1/ 1 finisher > > 1/ 1 reserver > > 1/ 5 heartbeatmap > > 1/ 5 perfcounter > > 1/ 5 rgw > > 1/ 5 rgw_sync > > 1/10 civetweb > > 1/ 5 javaclient > > 1/ 5 asok > > 1/ 1 throttle > > 0/ 0 refs > > 1/ 5 xio > > 1/ 5 compressor > > 1/ 5 bluestore > > 1/ 5 bluefs > > 1/ 3 bdev > > 1/ 5 kstore > > 4/ 5 rocksdb > > 4/ 5 leveldb > > 4/ 5 memdb > > 1/ 5 kinetic > > 1/ 5 fuse > > 1/ 5 mgr > > 1/ 5 mgrc > > 1/ 5 dpdk > > 1/ 5 eventtrace > > 1/ 5 prioritycache > > -2/-2 (syslog threshold) > > -1/-1 (stderr threshold) > > max_recent 10000 > > max_new 1000 > > log_file /var/log/ceph/ceph-osd.299.log > > --- end dump of recent events --- > > > > > > [2] > > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > > { > > "num_missing": 1, > > "num_unfound": 1, > > "objects": [ > > { > > "oid": { > > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > > "key": "", > > "snapid": -2, > > "hash": 12827, > > "max": 0, > > "pool": 36, > > "namespace": ".ceph-internal" > > }, > > "need": "1352209'2834660", > > "have": "0'0", > > "flags": "none", > > "locations": [] > > } > > ], > > "more": false > > } > > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > > { > > "num_missing": 1, > > "num_unfound": 1, > > "objects": [ > > { > > "oid": { > > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > > "key": "", > > "snapid": -2, > > "hash": 12874, > > "max": 0, > > "pool": 36, > > "namespace": ".ceph-internal" > > }, > > "need": "1361100'2822063", > > "have": "0'0", > > "flags": "none", > > "locations": [] > > } > > ], > > "more": false > > } > > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > > { > > "num_missing": 1, > > "num_unfound": 1, > > "objects": [ > > { > > "oid": { > > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > > "key": "", > > "snapid": -2, > > "hash": 4316, > > "max": 0, > > "pool": 36, > > "namespace": ".ceph-internal" > > }, > > "need": "1361089'2838543", > > "have": "0'0", > > "flags": "none", > > "locations": [] > > } > > ], > > "more": false > > } > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 3

ceph prometheus module no export content
by 黄明友 27 Feb '20

27 Feb '20

hi,all I had enable the prometheus module on my ceph cluster , the ceph version is ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable). when I eanble this modle, can get exported content from the prometheus module. but , when I restart all the ceph cluster's node , the module can't expor any content , curl http://host/metrics ; get blan; the tcp/9283 port is listen; I had disable and enable this module,but still no content export. 黄明友 IT基础架构部经理 V.Photos 云摄影移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy(a)v.photos 官方网址: www.v.photos 上海黄浦区中山东二路88号外滩SOHO3Q F栋 2层北京朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层广州天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区深圳南山区蛇口网谷科技大厦二期A座102网谷双创街 1层成都成华区建设路世贸广场 7层

4 4

Re: Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Paul Emmerich 27 Feb '20

27 Feb '20

I've also encountered this issue, but luckily without the crashing OSDs, so marking as lost resolved it for us. See https://tracker.ceph.com/issues/44286 Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Feb 27, 2020 at 6:02 AM Lincoln Bryant <lincolnb(a)uchicago.edu> wrote: > > Hello Ceph experts, > > In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. > > It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. > > Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. > > I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. > > Thanks, > Lincoln Bryant > > > [1] > -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none > -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete > -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] > -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] > 2: (()+0x4dddd7) [0x55955fdafdd7] > 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > 15: (()+0x7e25) [0x7ff5669bae25] > 16: (clone()+0x6d) [0x7ff565a9a34d] > > 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** > in thread 7ff52edaa700 thread_name:tp_osd_tp > > ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) > 1: (()+0xf5e0) [0x7ff5669c25e0] > 2: (gsignal()+0x37) [0x7ff5659d71f7] > 3: (abort()+0x148) [0x7ff5659d88e8] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] > 5: (()+0x4dddd7) [0x55955fdafdd7] > 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] > 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] > 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] > 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] > 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] > 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] > 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] > 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] > 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] > 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] > 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] > 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] > 18: (()+0x7e25) [0x7ff5669bae25] > 19: (clone()+0x6d) [0x7ff565a9a34d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 1/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 0 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 1 reserver > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 rgw_sync > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 4/ 5 memdb > 1/ 5 kinetic > 1/ 5 fuse > 1/ 5 mgr > 1/ 5 mgrc > 1/ 5 dpdk > 1/ 5 eventtrace > 1/ 5 prioritycache > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.299.log > --- end dump of recent events --- > > > [2] > [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound > { > "num_missing": 1, > "num_unfound": 1, > "objects": [ > { > "oid": { > "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", > "key": "", > "snapid": -2, > "hash": 12827, > "max": 0, > "pool": 36, > "namespace": ".ceph-internal" > }, > "need": "1352209'2834660", > "have": "0'0", > "flags": "none", > "locations": [] > } > ], > "more": false > } > [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound > { > "num_missing": 1, > "num_unfound": 1, > "objects": [ > { > "oid": { > "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", > "key": "", > "snapid": -2, > "hash": 12874, > "max": 0, > "pool": 36, > "namespace": ".ceph-internal" > }, > "need": "1361100'2822063", > "have": "0'0", > "flags": "none", > "locations": [] > } > ], > "more": false > } > [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound > { > "num_missing": 1, > "num_unfound": 1, > "objects": [ > { > "oid": { > "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", > "key": "", > "snapid": -2, > "hash": 4316, > "max": 0, > "pool": 36, > "namespace": ".ceph-internal" > }, > "need": "1361089'2838543", > "have": "0'0", > "flags": "none", > "locations": [] > } > ], > "more": false > } > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Is a scrub error (read_error) on a primary osd safe to repair?
by Mehmet 27 Feb '20

27 Feb '20

Hello Casper, did you found an answer on this topic? my guess is, that with "ceph pg repair" the copy of primary osd will overwrite the 2nd and 3rd - in case it is readable.. but what is when it is not readable? :thinking: Would be nice to know if there is a way to tell ceph to repair pg with copy from osd X. best regards, Mehmet Am 04.12.2019 um 13:47 schrieb Caspar Smit: > Hi all, > > I tried to dig in the mailinglist archives but couldn't find a clear > answer to the following situation: > > Ceph encountered a scrub error resulting in HEALTH_ERR > Two PG's are active+clean+inconsistent. When investigating the PG i see > a "read_error" on the primary OSD. Both PG's are replicated PG's with 3 > copies. > > I'm on Luminous 12.2.5 on this installation, is it safe to just run > "ceph pg repair" on those PG's or will it then overwrite the two good > copies with the bad one from the primary? > If the latter is true, what is the correct way to resolve this? > > Kind regards, > Caspar Smit > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >

2 1

Cache tier OSDs crashing due to unfound hitset object 14.2.7
by Lincoln Bryant 27 Feb '20

27 Feb '20

Hello Ceph experts, In the last day or so, we had a few nodes randomly reboot and now unfound objects are reported in Ceph health during cluster during recovery. It appears that the object in question is a hit set object, which I now cannot mark lost because Ceph cannot probe the OSDs that keep crashing due to missing the hit set object. Pasted below is the crash message[1] for osd.299, and some of the unfound objects[2]. Lastly [3] shows a sample of the hit set objects that are lost. I would greatly appreciate any insight you may have on how to move forward. As of right now this cluster is inoperable due to 3 down PGs. Thanks, Lincoln Bryant [1] -4> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1352209'2834660 flags = none -3> 2020-02-26 22:26:29.455 7ff52edaa700 0 0x559587fa91e0 36.321b unexpected need for 36:d84c0000:.ceph-internal::hit_set_36.321b_archive_2020-02-24 21%3a15%3a16.792846_2020-02-24 21%3a15%3a32.457855:head have 1352209'2834660 flags = none tried to add 1359781'2835659 flags = delete -2> 2020-02-26 22:26:29.456 7ff53adc2700 3 osd.299 1367392 handle_osd_map epochs [1367392,1367392], i have 1367392, src has [1349017,1367392] -1> 2020-02-26 22:26:29.460 7ff52edaa700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: In function 'void PG::MissingLoc::add_active_missing(const pg_missing_t&)' thread 7ff52edaa700 time 2020-02-26 22:26:29.457170 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/PG.h: 838: FAILED ceph_assert(i->second.need == j->second.need) ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55955fdafc0f] 2: (()+0x4dddd7) [0x55955fdafdd7] 3: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] 4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] 5: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] 6: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] 7: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] 8: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] 9: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] 15: (()+0x7e25) [0x7ff5669bae25] 16: (clone()+0x6d) [0x7ff565a9a34d] 0> 2020-02-26 22:26:29.465 7ff52edaa700 -1 *** Caught signal (Aborted) ** in thread 7ff52edaa700 thread_name:tp_osd_tp ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (()+0xf5e0) [0x7ff5669c25e0] 2: (gsignal()+0x37) [0x7ff5659d71f7] 3: (abort()+0x148) [0x7ff5659d88e8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55955fdafc5e] 5: (()+0x4dddd7) [0x55955fdafdd7] 6: (PG::MissingLoc::add_active_missing(pg_missing_set<false> const&)+0x1e0) [0x55955ffa0cb0] 7: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>, std::allocator<std::pair<int const, std::map<spg_t, pg_query_t, std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > > > > >&, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > >, std::less<int>, std::allocator<std::pair<int const, std::vector<std::pair<pg_notify_t, PastIntervals>, std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*, PG::RecoveryCtx*)+0x1916) [0x55955ff3f1e6] 8: (PG::RecoveryState::Active::Active(boost::statechart::state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::my_context)+0x370) [0x55955ff62d20] 9: (boost::statechart::simple_state<PG::RecoveryState::Peering, PG::RecoveryState::Primary, PG::RecoveryState::GetInfo, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xfb) [0x55955ffa8d5b] 10: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0x97) [0x55955ff88507] 11: (PG::handle_activate_map(PG::RecoveryCtx*)+0x1a8) [0x55955ff75848] 12: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x61d) [0x55955feb161d] 13: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa6) [0x55955feb2d16] 14: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55956011a481] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55955fea7bbf] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x559560448976] 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55956044b490] 18: (()+0x7e25) [0x7ff5669bae25] 19: (clone()+0x6d) [0x7ff565a9a34d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.299.log --- end dump of recent events --- [2] [root@ceph-mon01 ~]# ceph pg 36.321b list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "hit_set_36.321b_archive_2020-02-24 21:15:16.792846_2020-02-24 21:15:32.457855", "key": "", "snapid": -2, "hash": 12827, "max": 0, "pool": 36, "namespace": ".ceph-internal" }, "need": "1352209'2834660", "have": "0'0", "flags": "none", "locations": [] } ], "more": false } [root@ceph-mon01 ~]# ceph pg 36.324a list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "hit_set_36.324a_archive_2020-02-25 12:40:58.130723_2020-02-25 12:46:25.260587", "key": "", "snapid": -2, "hash": 12874, "max": 0, "pool": 36, "namespace": ".ceph-internal" }, "need": "1361100'2822063", "have": "0'0", "flags": "none", "locations": [] } ], "more": false } [root@ceph-mon01 ~]# ceph pg 36.10dc list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "hit_set_36.10dc_archive_2020-02-25 12:40:58.129048_2020-02-25 12:45:02.202268", "key": "", "snapid": -2, "hash": 4316, "max": 0, "pool": 36, "namespace": ".ceph-internal" }, "need": "1361089'2838543", "have": "0'0", "flags": "none", "locations": [] } ], "more": false }

1 0

Nautilus OSD memory consumption?
by Nigel Williams 27 Feb '20

27 Feb '20

The OOM-killer is on the rampage and striking down hapless OSDs when the cluster is under heavy client IO. The memory target does not seem to be much of a limit, is this intentional? root@cnx-11:~# ceph-conf --show-config|fgrep osd_memory_target osd_memory_target = 4294967296 osd_memory_target_cgroup_limit_ratio = 0.800000 root@cnx-31:~# pmap 4327|fgrep total total 6794892K Are there any tips for controlling the OSD memory consumption? The hosts involved have 128GB or 192GB memory, 12 x OSDs (SATA), so even with 4GB per OSD there should be a large amount of free memory.

2 3

default data pools for cephfs: replicated vs. ec
by thoralf schulze 26 Feb '20

26 Feb '20

hi there, recently, we've come across a lot of advice to only use replicated rados pools as default- (ie: root-) data pools for cephfs¹. unfortunately, we either skipped or blatantly ignored this advice while creating our cephfs, so our default data pool is an erasure coded one with k=2 and m=4, which _should_ be fine availability-wise. could anyone elaborate on the impacts regarding the performance of the whole setup? if a migration to a replicated pool is recommend: would a simple ceph osd pool set $default_data crush_rule $something_replicated suffice, or would you recommend a more elaborated approach, something along the lines of taking the cephfs down, copy contents of default_pool to default_new, rename default_new default_pool, taking the cephfs up again? thank you very much & with kind regards, t. ¹ - see, for instance, https://tracker.ceph.com/issues/42450 .

2 1

radosgw lifecycle seems work strangely
by quexian da 26 Feb '20

26 Feb '20

ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) I made a bucket named "test_lc" and ran `s3cmd expire --expiry-date=2019-01-01 s3://test_lc` to set the lifecycle (2019-01-01 is earlier than current date so every object will be removed). Then I ran `radosgw-admin lc process`, the objects got deleted as expected, and the status from `radosgw-admin lc list` is "completed". However, if I upload some objects, and ran `radosgw-admin lc process` again, the objects were not deleted. Could you please tell me what the reason is and what I should do in this case? Thanks in advance!

2 1

cepfs: ceph-fuse clients getting stuck + causing degraded PG
by Andras Pataki 26 Feb '20

26 Feb '20

We've been running into a strange problem repeating every day or so with a specific HPC job on a Mimic cluster (13.2.8) using ceph-fuse (14.2.7). It seems like some cephfs clients are stuck (perhaps deadlocked) trying to access a file and are not making progress. Ceph reports the following problems (ignore the oversized cache, large omap objects and the failing to respond to cache pressure - we get those regularly without any problems): [root@cephmon00 ~]# ceph -s cluster: id: d7b33135-0940-4e48-8aa6-1d2026597c2f health: HEALTH_WARN 1 MDSs report oversized cache 1 clients failing to respond to capability release 5 clients failing to respond to cache pressure 1 MDSs report slow requests 2 large omap objects Degraded data redundancy: 1/24563062389 objects degraded (0.000%), 1 pg degraded 1 slow ops, oldest one blocked for 14723 sec, osd.1058 has slow ops services: mon: 5 daemons, quorum cephmon00,cephmon01,cephmon02,cephmon03,cephmon04 mgr: cephmon01(active), standbys: cephmon02, cephmon03, cephmon04, cephmon00 mds: cephfs-1/1/1 up {0=cephmds00=up:active}, 5 up:standby osd: 3264 osds: 3264 up, 3264 in data: pools: 4 pools, 67072 pgs objects: 3.10 G objects, 10 PiB usage: 17 PiB used, 6.0 PiB / 23 PiB avail pgs: 1/24563062389 objects degraded (0.000%) 67006 active+clean 40 active+clean+scrubbing+deep 25 active+clean+scrubbing 1 active+recovering+degraded io: client: 501 MiB/s rd, 4.2 GiB/s wr, 1.63 kop/s rd, 6.46 kop/s wr recovery: 28 MiB/s, 7 objects/s [root@cephmon00 ~]# ceph health detail HEALTH_WARN 1 MDSs report oversized cache; 1 clients failing to respond to capability release; 5 clients failing to respond to cache pressure; 1 MDSs report slow requests; 2 large omap objects; Degraded data redundancy: 1/24563067054 objects degraded (0.000%), 1 pg degraded; 1 slow ops, oldest one blocked for 14723 sec, osd.1058 has slow ops MDS_CACHE_OVERSIZED 1 MDSs report oversized cache mdscephmds00(mds.0): MDS cache is too large (73GB/16GB); 27115370 inodes in use by clients, 3811 stray files MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdscephmds00(mds.0): Client worker1003:cephfs failing to respond to capability release client_id: 90142364 MDS_CLIENT_RECALL 5 clients failing to respond to cache pressure mdscephmds00(mds.0): Client gw162.flatironinstitute.org failing to respond to cache pressure client_id: 72550178 mdscephmds00(mds.0): Client ccblin053 failing to respond to cache pressure client_id: 76380719 mdscephmds00(mds.0): Client rusty2 failing to respond to cache pressure client_id: 79497905 mdscephmds00(mds.0): Client infolin003:cephfs failing to respond to cache pressure client_id: 88565901 mdscephmds00(mds.0): Client ceph-nfs1.flatironinstitute.org:cephfs-ro failing to respond to cache pressure client_id: 88899914 MDS_SLOW_REQUEST 1 MDSs report slow requests mdscephmds00(mds.0): 7 slow requests are blocked > 30 secs LARGE_OMAP_OBJECTS 2 large omap objects 2 large objects found in pool 'cephfs_metadata' Search the cluster log for 'Large omap object found' for more details. PG_DEGRADED Degraded data redundancy: 1/24563067054 objects degraded (0.000%), 1 pg degraded pg 2.14b7 is active+recovering+degraded, acting [1058,2098,734] SLOW_OPS 1 slow ops, oldest one blocked for 14723 sec, osd.1058 has slow ops The client that is failing to respond to capability release seems to be the real problem here. This is a ceph-fuse client (14.2.7) - it looks like it has a bunch of open requests in flight to the MDS: [root@worker1003 ~]# ceph daemon /var/run/ceph/ceph-client.cephfs.6969.93825141492560.asok mds_requests { "request": { "tid": 3801617, "op": "open", "path": "#0x10048b87a3b", "path2": "", "ino": "0x10048b87a3b", "hint_ino": "0x0", "sent_stamp": "2020-02-26 03:42:58.359939", "mds": 0, "resend_mds": -1, "send_to_auth": 0, "sent_on_mseq": 0, "retry_attempt": 1, "got_unsafe": 0, "uid": 1261, "gid": 1261, "oldest_client_tid": 3801617, "mdsmap_epoch": 0, "flags": 0, "num_retry": 0, "num_fwd": 0, "num_releases": 0, "abort_rc": 0 }, "request": { "tid": 3900154, "op": "lookup", "path": "#0x10048b61f09/tesscut_20200226034255.zip", "path2": "", "ino": "0x10048b61f09", "hint_ino": "0x0", "sent_stamp": "2020-02-26 04:08:50.572111", "mds": 0, "resend_mds": -1, "send_to_auth": 0, "sent_on_mseq": 0, "retry_attempt": 1, "got_unsafe": 0, "uid": 1261, "gid": 1261, "oldest_client_tid": 3801617, "mdsmap_epoch": 0, "flags": 0, "num_retry": 0, "num_fwd": 0, "num_releases": 0, "abort_rc": 0 }, "request": { "tid": 4111017, "op": "lookup", "path": "#0x10048b61f09/tesscut_20200226034255.zip", "path2": "", "ino": "0x10048b61f09", "hint_ino": "0x0", "sent_stamp": "2020-02-26 04:53:29.372694", "mds": 0, "resend_mds": -1, "send_to_auth": 0, "sent_on_mseq": 0, "retry_attempt": 1, "got_unsafe": 0, "uid": 1261, "gid": 1261, "oldest_client_tid": 3801617, "mdsmap_epoch": 0, "flags": 0, "num_retry": 0, "num_fwd": 0, "num_releases": 0, "abort_rc": 0 }, "request": { "tid": 4288428, "op": "lookup", "path": "#0x10048b61f09/tesscut_20200226034255.zip", "path2": "", "ino": "0x10048b61f09", "hint_ino": "0x0", "sent_stamp": "2020-02-26 05:39:52.257977", "mds": 0, "resend_mds": -1, "send_to_auth": 0, "sent_on_mseq": 0, "retry_attempt": 1, "got_unsafe": 0, "uid": 1261, "gid": 1261, "oldest_client_tid": 3801617, "mdsmap_epoch": 0, "flags": 0, "num_retry": 0, "num_fwd": 0, "num_releases": 0, "abort_rc": 0 }, "request": { "tid": 4425681, "op": "lookup", "path": "#0x10048b61f09/tesscut_20200226034255.zip", "path2": "", "ino": "0x10048b61f09", "hint_ino": "0x0", "sent_stamp": "2020-02-26 06:23:52.201814", "mds": 0, "resend_mds": -1, "send_to_auth": 0, "sent_on_mseq": 0, "retry_attempt": 1, "got_unsafe": 0, "uid": 1261, "gid": 1261, "oldest_client_tid": 3801617, "mdsmap_epoch": 0, "flags": 0, "num_retry": 0, "num_fwd": 0, "num_releases": 0, "abort_rc": 0 } } What seems to be odd is that the last 4 are for the same file. The MDS also sees these ops as pending: [root@cephmds00 ~]# ceph daemon /var/run/ceph/ceph-mds.cephmds00.asok ops { "ops": [ { "description": "client_request(client.90142364:4288428 lookup #0x10048b61f09/tesscut_20200226034255.zip 2020-02-26 05:39:52.257975 caller_uid=1261, caller_gid=1261{1193,1207,1223,1261,})", "initiated_at": "2020-02-26 05:39:52.258280", "age": 8189.169056, "duration": 8189.169070, "type_data": { "flag_point": "failed to rdlock, waiting", "reqid": "client.90142364:4288428", "op_type": "client_request", "client_info": { "client": "client.90142364", "tid": 4288428 }, "events": [ { "time": "2020-02-26 05:39:52.258280", "event": "initiated" }, { "time": "2020-02-26 05:39:52.258280", "event": "header_read" }, { "time": "2020-02-26 05:39:52.258281", "event": "throttled" }, { "time": "2020-02-26 05:39:52.258283", "event": "all_read" }, { "time": "2020-02-26 05:39:52.258362", "event": "dispatched" }, { "time": "2020-02-26 05:39:52.258383", "event": "failed to rdlock, waiting" } ] } }, { "description": "client_request(client.90142364:4425681 lookup #0x10048b61f09/tesscut_20200226034255.zip 2020-02-26 06:23:52.201811 caller_uid=1261, caller_gid=1261{1193,1207,1223,1261,})", "initiated_at": "2020-02-26 06:23:52.201698", "age": 5549.225638, "duration": 5549.225679, "type_data": { "flag_point": "failed to rdlock, waiting", "reqid": "client.90142364:4425681", "op_type": "client_request", "client_info": { "client": "client.90142364", "tid": 4425681 }, "events": [ { "time": "2020-02-26 06:23:52.201698", "event": "initiated" }, { "time": "2020-02-26 06:23:52.201698", "event": "header_read" }, { "time": "2020-02-26 06:23:52.201699", "event": "throttled" }, { "time": "2020-02-26 06:23:52.201702", "event": "all_read" }, { "time": "2020-02-26 06:23:52.201720", "event": "dispatched" }, { "time": "2020-02-26 06:23:52.201740", "event": "failed to rdlock, waiting" } ] } }, { "description": "client_request(client.90142364:4111017 lookup #0x10048b61f09/tesscut_20200226034255.zip 2020-02-26 04:53:29.372691 caller_uid=1261, caller_gid=1261{1193,1207,1223,1261,})", "initiated_at": "2020-02-26 04:53:29.373246", "age": 10972.054090, "duration": 10972.054155, "type_data": { "flag_point": "failed to rdlock, waiting", "reqid": "client.90142364:4111017", "op_type": "client_request", "client_info": { "client": "client.90142364", "tid": 4111017 }, "events": [ { "time": "2020-02-26 04:53:29.373246", "event": "initiated" }, { "time": "2020-02-26 04:53:29.373246", "event": "header_read" }, { "time": "2020-02-26 04:53:29.373246", "event": "throttled" }, { "time": "2020-02-26 04:53:29.373249", "event": "all_read" }, { "time": "2020-02-26 04:53:29.373299", "event": "dispatched" }, { "time": "2020-02-26 04:53:29.373323", "event": "failed to rdlock, waiting" } ] } }, { "description": "client_request(client.90118987:32628772 lookup #0x10048b61f09/tesscut_20200226034255.zip 2020-02-26 04:09:03.139173 caller_uid=1261, caller_gid=1261{1193,1207,1223,1261,})", "initiated_at": "2020-02-26 04:09:03.138539", "age": 13638.288797, "duration": 13638.288884, "type_data": { "flag_point": "failed to rdlock, waiting", "reqid": "client.90118987:32628772", "op_type": "client_request", "client_info": { "client": "client.90118987", "tid": 32628772 }, "events": [ { "time": "2020-02-26 04:09:03.138539", "event": "initiated" }, { "time": "2020-02-26 04:09:03.138539", "event": "header_read" }, { "time": "2020-02-26 04:09:03.138540", "event": "throttled" }, { "time": "2020-02-26 04:09:03.138543", "event": "all_read" }, { "time": "2020-02-26 04:09:03.138569", "event": "dispatched" }, { "time": "2020-02-26 04:09:03.138592", "event": "failed to rdlock, waiting" } ] } }, { "description": "client_request(client.90142364:3801617 open #0x10048b87a3b 2020-02-26 03:42:58.359934 caller_uid=1261, caller_gid=1261{1193,1207,1223,1261,})", "initiated_at": "2020-02-26 03:42:58.359502", "age": 15203.067834, "duration": 15203.067944, "type_data": { "flag_point": "failed to xlock, waiting", "reqid": "client.90142364:3801617", "op_type": "client_request", "client_info": { "client": "client.90142364", "tid": 3801617 }, "events": [ { "time": "2020-02-26 03:42:58.359502", "event": "initiated" }, { "time": "2020-02-26 03:42:58.359502", "event": "header_read" }, { "time": "2020-02-26 03:42:58.359502", "event": "throttled" }, { "time": "2020-02-26 03:42:58.359505", "event": "all_read" }, { "time": "2020-02-26 03:42:58.359525", "event": "dispatched" }, { "time": "2020-02-26 03:42:58.359557", "event": "failed to xlock, waiting" } ] } }, { "description": "client_request(client.90145864:3374743 lookup #0x10048b61f09/tesscut_20200226034255.zip 2020-02-26 06:24:02.082413 caller_uid=1261, caller_gid=1261{1193,1207,1223,1261,})", "initiated_at": "2020-02-26 06:24:02.081762", "age": 5539.345573, "duration": 5539.345708, "type_data": { "flag_point": "failed to rdlock, waiting", "reqid": "client.90145864:3374743", "op_type": "client_request", "client_info": { "client": "client.90145864", "tid": 3374743 }, "events": [ { "time": "2020-02-26 06:24:02.081762", "event": "initiated" }, { "time": "2020-02-26 06:24:02.081762", "event": "header_read" }, { "time": "2020-02-26 06:24:02.081763", "event": "throttled" }, { "time": "2020-02-26 06:24:02.081766", "event": "all_read" }, { "time": "2020-02-26 06:24:02.081787", "event": "dispatched" }, { "time": "2020-02-26 06:24:02.081804", "event": "failed to rdlock, waiting" } ] } }, { "description": "client_request(client.90142364:3900154 lookup #0x10048b61f09/tesscut_20200226034255.zip 2020-02-26 04:08:50.572109 caller_uid=1261, caller_gid=1261{1193,1207,1223,1261,})", "initiated_at": "2020-02-26 04:08:50.572289", "age": 13650.855047, "duration": 13650.855205, "type_data": { "flag_point": "failed to rdlock, waiting", "reqid": "client.90142364:3900154", "op_type": "client_request", "client_info": { "client": "client.90142364", "tid": 3900154 }, "events": [ { "time": "2020-02-26 04:08:50.572289", "event": "initiated" }, { "time": "2020-02-26 04:08:50.572289", "event": "header_read" }, { "time": "2020-02-26 04:08:50.572290", "event": "throttled" }, { "time": "2020-02-26 04:08:50.572293", "event": "all_read" }, { "time": "2020-02-26 04:08:50.572318", "event": "dispatched" }, { "time": "2020-02-26 04:08:50.572338", "event": "failed to rdlock, waiting" } ] } } ], "num_ops": 7 } There is another client on this list (90118987) also trying to access the same file - here is the client's op list from that (worker1020): [root@worker1020 ~]# ceph daemon /var/run/ceph/ceph-client.cephfs.6958.93825141492560.asok mds_requests { "request": { "tid": 32628772, "op": "lookup", "path": "#0x10048b61f09/tesscut_20200226034255.zip", "path2": "", "ino": "0x10048b61f09", "hint_ino": "0x0", "sent_stamp": "2020-02-26 04:09:03.139177", "mds": 0, "resend_mds": -1, "send_to_auth": 0, "sent_on_mseq": 0, "retry_attempt": 1, "got_unsafe": 0, "uid": 1261, "gid": 1261, "oldest_client_tid": 32628772, "mdsmap_epoch": 0, "flags": 0, "num_retry": 0, "num_fwd": 0, "num_releases": 0, "abort_rc": 0 } } Also, the kernel (CentOS 7 3.10.0-1062.9.1.el7.x86_64) also reports stuck processes. dmesg: [539762.054205] INFO: task python:1098178 blocked for more than 120 seconds. [539762.054210] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054212] python D ffff8835f8788640 0 1098178 1098093 0x00000000 [539762.054218] Call Trace: [539762.054252] [<ffffffffc09abfcf>] ? fuse_do_open+0x14f/0x1e0 [fuse] [539762.054259] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054263] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054269] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054274] [<ffffffff95d28df6>] ima_file_check+0xa6/0x1b0 [539762.054280] [<ffffffff95c592ca>] do_last+0x59a/0x1290 [539762.054284] [<ffffffff95c5bdbd>] path_openat+0xcd/0x5a0 [539762.054290] [<ffffffff95c4737a>] ? __check_object_size+0x1ca/0x250 [539762.054294] [<ffffffff95c5d72d>] do_filp_open+0x4d/0xb0 [539762.054299] [<ffffffff95c6b207>] ? __alloc_fd+0x47/0x170 [539762.054303] [<ffffffff95c49684>] do_sys_open+0x124/0x220 [539762.054307] [<ffffffff95c4979e>] SyS_open+0x1e/0x20 [539762.054311] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054317] INFO: task python:1098179 blocked for more than 120 seconds. [539762.054318] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054320] python D ffff88847f25ac80 0 1098179 1098093 0x00000000 [539762.054324] Call Trace: [539762.054328] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054331] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054335] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054344] [<ffffffffc09aeb3b>] fuse_file_aio_write+0xdb/0x390 [fuse] [539762.054348] [<ffffffff95c49d03>] do_sync_write+0x93/0xe0 [539762.054353] [<ffffffff95c4a7f0>] vfs_write+0xc0/0x1f0 [539762.054357] [<ffffffff95c4b60f>] SyS_write+0x7f/0xf0 [539762.054360] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054366] INFO: task python:1098180 blocked for more than 120 seconds. [539762.054368] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054369] python D ffff88847f25ac80 0 1098180 1098093 0x00000000 [539762.054372] Call Trace: [539762.054376] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054379] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054384] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054392] [<ffffffffc09aeb3b>] fuse_file_aio_write+0xdb/0x390 [fuse] [539762.054398] [<ffffffff95bf1249>] ? handle_pte_fault+0x2b9/0xe20 [539762.054402] [<ffffffff95c49d03>] do_sync_write+0x93/0xe0 [539762.054406] [<ffffffff95c4a7f0>] vfs_write+0xc0/0x1f0 [539762.054410] [<ffffffff95c4b60f>] SyS_write+0x7f/0xf0 [539762.054413] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054420] INFO: task python:1098181 blocked for more than 120 seconds. [539762.054421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054423] python D ffff88847f45ac80 0 1098181 1098093 0x00000004 [539762.054426] Call Trace: [539762.054430] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054433] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054438] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054446] [<ffffffffc09ac5e6>] fuse_flush+0xb6/0x200 [fuse] [539762.054452] [<ffffffff96188678>] ? __do_page_fault+0x238/0x500 [539762.054455] [<ffffffff95c47ce7>] filp_close+0x37/0x90 [539762.054458] [<ffffffff95c6b4ac>] __close_fd+0x8c/0xb0 [539762.054461] [<ffffffff95c49803>] SyS_close+0x23/0x50 [539762.054465] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054482] INFO: task python:1098186 blocked for more than 120 seconds. [539762.054483] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054485] python D ffff88847f45ac80 0 1098186 1098093 0x00000000 [539762.054488] Call Trace: [539762.054492] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054494] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054499] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054507] [<ffffffffc09ac5e6>] fuse_flush+0xb6/0x200 [fuse] [539762.054511] [<ffffffff95c47ce7>] filp_close+0x37/0x90 [539762.054514] [<ffffffff95c6b4ac>] __close_fd+0x8c/0xb0 [539762.054517] [<ffffffff95c49803>] SyS_close+0x23/0x50 [539762.054520] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054529] INFO: task python:1098188 blocked for more than 120 seconds. [539762.054531] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054532] python D ffff88847f39ac80 0 1098188 1098093 0x00000004 [539762.054536] Call Trace: [539762.054539] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054542] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054546] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054554] [<ffffffffc09ac5e6>] fuse_flush+0xb6/0x200 [fuse] [539762.054559] [<ffffffff96188678>] ? __do_page_fault+0x238/0x500 [539762.054562] [<ffffffff95c47ce7>] filp_close+0x37/0x90 [539762.054565] [<ffffffff95c6b4ac>] __close_fd+0x8c/0xb0 [539762.054568] [<ffffffff95c49803>] SyS_close+0x23/0x50 [539762.054572] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054577] INFO: task python:1098189 blocked for more than 120 seconds. [539762.054579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054580] python D ffff88847f29ac80 0 1098189 1098093 0x00000004 [539762.054584] Call Trace: [539762.054587] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054590] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054595] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054602] [<ffffffffc09ac5e6>] fuse_flush+0xb6/0x200 [fuse] [539762.054606] [<ffffffff95c47ce7>] filp_close+0x37/0x90 [539762.054609] [<ffffffff95c6b4ac>] __close_fd+0x8c/0xb0 [539762.054612] [<ffffffff95c49803>] SyS_close+0x23/0x50 [539762.054615] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054626] INFO: task python:1098192 blocked for more than 120 seconds. [539762.054628] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054630] python D ffff88847f31ac80 0 1098192 1098093 0x00000004 [539762.054633] Call Trace: [539762.054636] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054639] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054644] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054651] [<ffffffffc09ac5e6>] fuse_flush+0xb6/0x200 [fuse] [539762.054656] [<ffffffff96188678>] ? __do_page_fault+0x238/0x500 [539762.054659] [<ffffffff95c47ce7>] filp_close+0x37/0x90 [539762.054662] [<ffffffff95c6b4ac>] __close_fd+0x8c/0xb0 [539762.054665] [<ffffffff95c49803>] SyS_close+0x23/0x50 [539762.054668] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539762.054694] INFO: task python:1098202 blocked for more than 120 seconds. [539762.054696] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539762.054697] python D ffff88847f49ac80 0 1098202 1098093 0x00000000 [539762.054701] Call Trace: [539762.054709] [<ffffffffc09abfcf>] ? fuse_do_open+0x14f/0x1e0 [fuse] [539762.054713] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539762.054715] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539762.054720] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539762.054723] [<ffffffff95d28df6>] ima_file_check+0xa6/0x1b0 [539762.054727] [<ffffffff95c592ca>] do_last+0x59a/0x1290 [539762.054731] [<ffffffff95c5bdbd>] path_openat+0xcd/0x5a0 [539762.054735] [<ffffffff95c4737a>] ? __check_object_size+0x1ca/0x250 [539762.054739] [<ffffffff95c5d72d>] do_filp_open+0x4d/0xb0 [539762.054744] [<ffffffff95c6b207>] ? __alloc_fd+0x47/0x170 [539762.054748] [<ffffffff95c49684>] do_sys_open+0x124/0x220 [539762.054751] [<ffffffff95c4979e>] SyS_open+0x1e/0x20 [539762.054755] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a [539882.054331] INFO: task python:1098178 blocked for more than 120 seconds. [539882.054334] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [539882.054335] python D ffff8835f8788640 0 1098178 1098093 0x00000004 [539882.054339] Call Trace: [539882.054367] [<ffffffffc09abfcf>] ? fuse_do_open+0x14f/0x1e0 [fuse] [539882.054371] [<ffffffff96181929>] schedule_preempt_disabled+0x29/0x70 [539882.054374] [<ffffffff9617f8b7>] __mutex_lock_slowpath+0xc7/0x1d0 [539882.054377] [<ffffffff9617ec8f>] mutex_lock+0x1f/0x2f [539882.054381] [<ffffffff95d28df6>] ima_file_check+0xa6/0x1b0 [539882.054385] [<ffffffff95c592ca>] do_last+0x59a/0x1290 [539882.054387] [<ffffffff95c5bdbd>] path_openat+0xcd/0x5a0 [539882.054391] [<ffffffff95c4737a>] ? __check_object_size+0x1ca/0x250 [539882.054394] [<ffffffff95c5d72d>] do_filp_open+0x4d/0xb0 [539882.054398] [<ffffffff95c6b207>] ? __alloc_fd+0x47/0x170 [539882.054400] [<ffffffff95c49684>] do_sys_open+0x124/0x220 [539882.054402] [<ffffffff95c4979e>] SyS_open+0x1e/0x20 [539882.054405] [<ffffffff9618dede>] system_call_fastpath+0x25/0x2a Now - independently from this there is a degraded PG in the data pool (NOT in the metadata pool interestingly enough) [root@cephmon00 ~]# ceph pg dump | grep degraded dumped all 2.14b7 16292 0 1 0 0 25530224980 0 0 3037 3037 active+recovering+degraded 2020-02-26 08:03:40.561062 1492172'6549513 1492172:10385376 [1058,2098,734] 1058 [1058,2098,734] 1058 1492086'6537090 2020-02-22 20:06:11.092352 1485071'6362109 2020-01-29 21:48:02.821467 0 [root@cephmon00 ~]# ceph osd pool ls detail pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 32768 pgp_num 32768 last_change 1484983 lfor 0/1355093 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application cephfs pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 512 pgp_num 512 last_change 1484989 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application cephfs pool 5 'rbd_vm' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 1487461 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd removed_snaps [1~3,5~2] pool 9 'cephfs_data_ec63' erasure size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 32768 pgp_num 32768 last_change 1484986 lfor 0/1355095 flags hashpspool,ec_overwrites stripe_width 24576 application cephfs osd.1058 keeps complaining about a missing object and crc mismatch: ... 2020-02-26 08:05:44.453 7fffcb3fe700 -1 log_channel(cluster) log [ERR] : 2.14b7 missing primary copy of 2:ed28cfe5:::10048b87a3b.00000001:head, will try copies on 734,2098 2020-02-26 08:05:44.585 7fffcb3fe700 -1 log_channel(cluster) log [ERR] : 2.14b7 full-object read crc 0x91584ffd != expected 0xbd4cf65d on 2:ed28cfe5:::10048b87a3b.00000001:head 2020-02-26 08:05:44.609 7fffcb3fe700 -1 log_channel(cluster) log [ERR] : 2.14b7 full-object read crc 0x91584ffd != expected 0xbd4cf65d on 2:ed28cfe5:::10048b87a3b.00000001:head 2020-02-26 08:05:44.609 7fffcb3fe700 -1 log_channel(cluster) log [ERR] : 2.14b7 missing primary copy of 2:ed28cfe5:::10048b87a3b.00000001:head, will try copies on 734,2098 2020-02-26 08:05:44.717 7fffcb3fe700 -1 log_channel(cluster) log [ERR] : 2.14b7 full-object read crc 0x91584ffd != expected 0xbd4cf65d on 2:ed28cfe5:::10048b87a3b.00000001:head 2020-02-26 08:05:44.741 7fffcb3fe700 -1 log_channel(cluster) log [ERR] : 2.14b7 full-object read crc 0x91584ffd != expected 0xbd4cf65d on 2:ed28cfe5:::10048b87a3b.00000001:head 2020-02-26 08:05:44.741 7fffcb3fe700 -1 log_channel(cluster) log [ERR] : 2.14b7 missing primary copy of 2:ed28cfe5:::10048b87a3b.00000001:head, will try copies on 734,2098 ... When I stop this OSD, one of its secondaries takes over and then THAT osd complains about the exact same issue. What is most bizarre about this is that if I power cycle the client nodes that are stuck - the whole issue resolves itself, including the degraded PG. So somehow these stuck ceph-fuse processes are responsible for this degraded PG as well. Not quite sure how. This issue happened 4 times in the past few days on different nodes - so it is definitely not a hardware issue. Are there any known issues we're running into here, or is this a new problem? Andras

1 0

Running MDS server on a newer version than monitoring nodes
by Martin Palma 26 Feb '20

26 Feb '20

Hi, is it possible to run MDS on a newer version than the monitoring nodes? I mean we run monitoring nodes on 12.2.10 and would like to upgrade the MDS to 12.2.13 is this possible? Best, Martin

2 2

Ceph standby-replay metadata server: MDS internal heartbeat is not healthy
by Martin Palma 26 Feb '20

26 Feb '20

Hi all, today we observe that out of the sudden our standby-replay metadata server continuously writes the following logs: 2020-02-13 11:56:50.216102 7fd2ad229700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-02-13 11:56:50.287699 7fd2ad229700 0 mds.beacon.dcucmds401 Skipping beacon heartbeat to monitors (last acked 100.836s ago); MDS internal heartbeat is not healthy! and it's memory is growing until no memory is available any more and the service gets restarted and then stops. The funny thing is that on the active MDS we are not seeing these log messages and any increase of memory. We are running ceph version 12.2.10 on all nodes of our Ceph cluster. Any suggestions? Best, Martin

2 2

next Ceph Meetup Berlin, Germany
by Robert Sander 26 Feb '20

26 Feb '20

Hi, The Ceph Berlin MeetUp is a community organized group that met bi-monthly in the past years: https://www.meetup.com/Ceph-Berlin/ The meetups start at 6 pm and consist of one presentation or talk and a following discussion. The discussion often takes place over dinner in a nearby restaurant if available. The next date would be March 23rd in four weeks, enough time to organize a meetup. Before fixing that date I would like to ask everyone if someone is willing to host our MeetUp. This is quite uncomplicated, we just need a room for up to 20 people and a beamer for a little talk. Catering is completely optional but very welcome. If March 23rd does not fit your schedule please suggest another day. So if you or your company in Berlin is able and willing to host the next Ceph meetup please contact me. If you have done something with Ceph in the last year and want to talk about it, please also do not hesitate to contact me. Kindest Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Frequest LARGE_OMAP_OBJECTS in cephfs metadata pool
by Uday Bhaskar jalagam 26 Feb '20

26 Feb '20

Hello Team , I am getting frequent LARGE_OMAP_OBJECTS 1 large omap objects in one of my cephfs metadata pools , anyone can explain why would this pool getting into this state frequently and how could I prevent this in future ? # ceph health detail HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'cephfs01-metadata' Search the cluster log for 'Large omap object found' for more details. Thanks , Uday

2 5

Limited performance
by Fabian Zimmermann 25 Feb '20

25 Feb '20

Hi, we currently creating a new cluster. This cluster is (as far as we can tell) an config-copy (ansible) of our existing cluster, just 5 years later - with new hardware (nvme instead of ssd, bigger disks, ...) The setup: * NVMe for Journals and "Cache"-Pool * HDD with NVMe Journals for "Data"-Pool * Cache-Pool as writeback-Tier on Data-Pool * We are using 12.2.13 without bluestore. If we run an rados benchmark against this pool, everything seems fine, but as soon as we start a fio-benchmark -<- [global] ioengine=rbd clientname=cinder pool=cinder rbdname=fio_test rw=write bs=4M [rbd_iodepth32] iodepth=32 ->- after some seconds the bandwidth drops to <15 MB/s and our hdd-disks are doing more IOs than our Journal-Disks. We also unconfigured the caching completely, but the issue remains. The output of "ceph osd pool stats" shows ~100 op/s, but our disks are doing: -<- Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 0.00 0.00 278.50 0.00 34.07 250.51 0.14 0.50 0.00 0.50 0.03 0.80 nvme1n1 0.00 0.00 0.00 64.00 0.00 7.77 248.50 0.01 0.22 0.00 0.22 0.03 0.20 sda 0.00 1.50 0.00 557.00 0.00 29.49 108.45 180.57 160.59 0.00 160.59 1.80 100.00 sdb 0.00 42.00 0.00 592.00 0.00 28.21 97.60 176.51 1105.79 0.00 1105.79 1.69 100.00 sdc 0.00 14.50 0.00 528.50 0.00 27.95 108.31 183.02 179.47 0.00 179.47 1.89 100.00 sde 0.00 134.50 0.00 223.50 0.00 14.05 128.72 17.38 60.05 0.00 60.05 0.89 20.00 sdg 0.00 76.00 0.00 492.00 0.00 26.32 109.54 191.81 1474.96 0.00 1474.96 2.03 100.00 sdf 0.00 0.00 0.00 491.50 0.00 26.76 111.49 176.55 326.05 0.00 326.05 2.03 100.00 sdh 0.00 0.00 0.00 548.50 0.00 26.71 99.75 204.39 327.57 0.00 327.57 1.82 100.00 sdi 0.00 112.00 0.00 526.00 0.00 23.15 90.14 158.32 1325.61 0.00 1325.61 1.90 100.00 sdj 0.00 12.00 0.00 641.00 0.00 34.78 111.13 185.51 278.29 0.00 278.29 1.56 100.00 sdk 0.00 23.50 0.00 399.50 0.00 20.38 104.46 166.77 461.67 0.00 461.67 2.50 100.00 sdl 0.00 267.00 0.00 498.50 0.00 34.46 141.58 200.37 490.80 0.00 490.80 2.01 100.00 ->- Any hints how to debug the issue? Thanks a lot, Fabian

3 2

Unable to increase PG numbers
by Andres Rojas Guerrero 25 Feb '20

25 Feb '20

Hi, I have a Nautilus installation version 14.2.1 with a very unbalanced cephfs pool, I have 430 osd in the cluster but this pool only have 8 PG and PGP and 118 TB used : # ceph -s cluster: id: a2269da7-e399-484a-b6ae-4ee1a31a4154 health: HEALTH_WARN 1 nearfull osd(s) 2 pool(s) nearfull services: mon: 3 daemons, quorum mon21,mon22,mon23 (age 7M) mgr: mon23(active, since 8M), standbys: mon22, mon21 mds: cephfs:2 {0=mon21=up:active,1=mon22=up:active} 1 up:standby osd: 430 osds: 430 up, 430 in data: pools: 2 pools, 16 pgs objects: 10.07M objects, 38 TiB usage: 118 TiB used, 4.5 PiB / 4.6 PiB avail pgs: 15 active+clean 1 active+clean+scrubbing+deep # ceph osd pool get cephfs_data pg_num pg_num: 8 Due to this bad configuration I have this warning message: # ceph status cluster: id: a2269da7-e399-484a-b6ae-4ee1a31a4154 health: HEALTH_WARN 1 nearfull osd(s) 2 pool(s) nearfull I've discover that some osd are full: # ceph osd status | 113 | osd23 | 9824G | 1351G | 0 | 0 | 0 | 0 | exists,nearfull,up | I've tried to reweight this osd: ceph osd reweight osd.113 0.9 But the process of reweight doesn't start. Otherwise I've tried to increase the PG and PGP numbers but it' doesn't work. # ceph osd pool set cephfs_data pg_num 128 set pool 1 pg_num to 128 # ceph osd pool get cephfs_data pg_num pg_num: 8 What could be the reason for this problem?

3 5

Migrating data to a more efficient EC pool
by Vladimir Brik 25 Feb '20

25 Feb '20

Hello I have ~300TB of data in default.rgw.buckets.data k2m2 pool and I would like to move it to a new k5m2 pool. I found instructions using cache tiering[1], but they come with a vague scary warning, and it looks like EC-EC may not even be possible [2] (is it still the case?). Can anybody recommend a safe procedure to copy an EC pool's data to another pool with a more efficient erasure coding? Perhaps there is a tool out there that could do it? A few days of downtime would be tolerable, if it will simplify things. Also, I have enough free space to temporarily store the k2m2 data in a replicated pool (if EC-EC tiering is not possible, but EC-replicated and replicated-EC tiering is possible). Is there a tool or some efficient way to verify that the content of two pools is the same? Thanks, Vlad [1] https://ceph.io/geen-categorie/ceph-pool-migration/ [2] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-February/016109.ht…

2 1

Changing allocation size
by Kristof Coucke 25 Feb '20

25 Feb '20

Hi all, A while back, I indicated we had an issue with our cluster filling up too fast. After checking everything, we've concluded this was because we had a lot of small files and the allocation size on the bluestore was too high (64kb). We are now recreating the OSD's (2 disk at the same time) but, this will take a very long time as we're dealing with 130 OSDs. The current process we're following is removing 2 osd's and recreating them. We're using erasure coding (6 + 3). Has anyone some advice on how we can move forward with this? We've already increased some parameters to speed up recovery, but even then, it would still cost us too much time. If we could recreate them faster, that would be great... Or adapt the allocation on the fly? Any suggestions are welcome... Thank you, Kristof.

2 1

unscheduled mds failovers
by danjou.philippe＠yahoo.de 25 Feb '20

25 Feb '20

Hi, we're seeing active mds failover to standbys every few weeks causing few minutes of cephfs downtime. It's not crashing, all the log says is. 2020-02-25 08:30:53.313 7f9a457ae700 1 mds.m2-1045557 Updating MDS map to version 10132 from mon.1 2020-02-25 08:30:53.313 7f9a457ae700 1 mds.m2-1045557 Map has assigned me to become a standby Anyone got any idea why that is? It's not so nice that ceph does it by itself. Thanks

1 0

Ceph @ SoCal Linux Expo
by Gregory Farnum 24 Feb '20

24 Feb '20

Hey all, we're excited to be returning properly to SCaLE in Pasadena[1] this year (March 5-8) with a Thursday Birds-of-a-Feather session[2] and a booth in the expo hall. Please come by if you're attending the conference or are in the area to get face time with other area users and Ceph developers. :) Also, I got drafted into organizing this so if you'd be willing to help man the booth in exchange for an Expo pass, shoot me an email! I think I've got 3 spots left. -Greg [1]: https://www.socallinuxexpo.org/scale/18x [2]: https://www.socallinuxexpo.org/scale/18x/presentations/ceph-storage

1 0

ceph-mon using 100% CPU after upgrade to 14.2.5
by Bryan Stillwell 24 Feb '20

24 Feb '20

On our test cluster after upgrading to 14.2.5 I'm having problems with the mons pegging a CPU core while moving data around. I'm currently converting the OSDs from FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the OSDs, and then recreating them with ceph-volume lvm batch. This seems too get the ceph-mon process into a state where it pegs a CPU core on one of the mons: 1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72 ceph-mon Has anyone else run into this with 14.2.5 yet? I didn't see this problem while the cluster was running 14.2.4. Thanks, Bryan

5 8

RGW do not show up in 'ceph status'
by Andreas Haupt 24 Feb '20

24 Feb '20

Dear all, we recently added two additional RGWs to our CEPH cluster (version 14.2.7). They work flawlessly, however they do not show up in 'ceph status': [cephmon1] /root # ceph -s | grep -A 6 services services: mon: 3 daemons, quorum cephmon1,cephmon2,cephmon3 (age 14h) mgr: cephmon1(active, since 14h), standbys: cephmon2, cephmon3 mds: cephfs:1 {0=cephmon1=up:active} 2 up:standby osd: 168 osds: 168 up (since 2w), 168 in (since 6w) rgw: 1 daemon active (ceph-s3) As you can see, only the first, old RGW (ceph-s3) is listed. Is there any place where the RGWs need to get "announced"? Any idea, how to debug this? Thanks, Andreas -- | Andreas Haupt | E-Mail: andreas.haupt(a)desy.de | DESY Zeuthen | WWW: http://www-zeuthen.desy.de/~ahaupt | Platanenallee 6 | Phone: +49/33762/7-7359 | D-15738 Zeuthen | Fax: +49/33762/7-7216

2 3

One PG is stuck and reading is not possible
by mikko.lampikoski＠doop.fi 24 Feb '20

24 Feb '20

ceph version 12.2.13 luminous (stable) My whole ceph cluster went to kind of read only state. Ceph status showed that client reads is 0 op/s for whole cluster. There was normal amount of writes going on. I checked health and it said: # ceph health detail HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg peering PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg peering pg 26.13b is stuck peering for 25523.506788, current state peering, last acting [2,0,33] All osds showed to be up and all monitors are good. All pools are 3/2 (size/min) and space usage ~30%. I fixed this by restarting forst osd.2 (nothing happened) and then restarted osd.0. After that everyting went back to normal. So what can cause "stuck peering" and how can i prevent this event from happening again?

1 0

Fwd: MDS Daemon Damaged
by Ben 23 Feb '20

23 Feb '20

I have a 3 node ceph cluster for my house that I have been using for a few years now without issue. Each node is a MON, MGR, and MDS, and has 2-3 OSDs on them. It has, however been slow. I decided to finally move the bluestore DBs to SSDs. I did one OSD as a test case to make sure everything was going to go OK. I deleted the OSD, then created a new OSD using the ceph-deploy tool and pointed the DB at a LVM partition on a SSD. Everything went OK, and recovery started. Later in the day I noticed that my MDS daemon is damaged (PGs are still recovering). I've tried the cephfs-journal-tool --rank=cephfs:all journal export backup.bin command, but it gave me: 2020-02-23 17:50:03.589 7f7d8b225740 -1 Missing object 200.00c30b6d 2020-02-23 17:50:07.919 7f7d8b225740 -1 Bad entry start ptr (0x30c2dbb92003) at 0x30c2d3a125ea (both lines have several repeats) and will not complete. Looking at the log file of the mds that was active at the time shows: 2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro) _finish_read got error -2 2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro) _finish_read got error -2 2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro) _finish_read got error -2 2020-02-23 17:13:09.091 7fad3e826700 0 mds.0.log _replay journaler got error -2, aborting 2020-02-23 17:13:09.091 7fad3e826700 -1 log_channel(cluster) log [ERR] : missing journal object One other thing that happened about the same time, I noticed I was having memory pressure on all the nodes with only 200MB of free ram. I've tweaked the bluestore osd_memory_target to try to help that not happen again. Even so, I'm a bit confused how that could cause catastrophic failure, as I had 2 other MDSes on standby. Any help would be appreciated.

1 0

Migrating/Realocating ceph cluster
by Rafał Wądołowski 21 Feb '20

21 Feb '20

Hi, I am looking for a good way of migrating/realocating ceph cluster. It has about 2PB net, mainly RBD, but object storage is also used. The new location is far away about 1,500 kilometers. Of course I have to minimize the downtime of the cluster :) Right now I see following scenarios: 1. Build identical cluster. Freeze the source. Copy everything with cppool/rbd mirror. Relocate servers and the power on them. 2. Run cluster mirroring over network. 3. Using cache tier. I'm starting research about this migration, so probably some of the solutions above are unfeasible. Maybe somebody has experience about migration between DC? Any new ideas? Thougths? Every comment will be helpful -- Regards, Rafał Wądołowski

3 4

Re: RESOLVED: Sudden loss of all SSD OSDs in a cluster, immedaite abort on restart [Mimic 13.2.6]
by Dan van der Ster 21 Feb '20

21 Feb '20

Hi Troy, Looks like we hit the same today -- Sage posted some observations here: https://tracker.ceph.com/issues/39525#note-6 Did it happen again in your cluster? Cheers, Dan On Tue, Aug 20, 2019 at 2:18 AM Troy Ablan <tablan(a)gmail.com> wrote: > > While I'm still unsure how this happened, this is what was done to solve > this. > > Started OSD in foreground with debug 10, watched for the most recent > osdmap epoch mentioned before abort(). For example, if it mentioned > that it just tried to load 80896 and then crashed > > # ceph osd getmap -o osdmap.80896 80896 > # ceph-objectstore-tool --op set-osdmap --data-path > /var/lib/ceph/osd/ceph-77/ --file osdmap.80896 > > Then I restarted the osd in foreground/debug, and repeated for the next > osdmap epoch until it got past the first few seconds. This process > worked for all but two OSDs. For the ones that succeeded I'd ^C and > then start the osd via systemd > > For the remaining two, it would try loading the incremental map and then > crash. I had presence of mind to make dd images of every OSD before > starting this process, so I reverted these two to the state before > injecting the osdmaps. > > I then injected the last 15 or so epochs of the osdmap in sequential > order before starting the osd, with success. > > This leads me to believe that the step-wise injection didn't work > because the osd had more subtle corruption that it got past, but it was > confused when it requested the next incremental delta. > > Thanks again to Brad/badone for the guidance! > > Tracker issue updated. > > Here's the closing IRC dialogue re this issue (UTC-0700) > > 2019-08-19 16:27:42 < MooingLemur> badone: I appreciate you reaching out > yesterday, you've helped a ton, twice now :) I'm still concerned > because I don't know how this happened. I'll feel better once > everything's active+clean, but it's all at least active. > > 2019-08-19 16:30:28 < badone> MooingLemur: I had a quick discussion with > Josh earlier and he shares my opinion this is likely somehow related to > these drives or perhaps controllers, or at least specific to these machines > > 2019-08-19 16:31:04 < badone> however, there is a possibility you are > seeing some extremely rare race that no one up to this point has seen before > > 2019-08-19 16:31:20 < badone> that is less likely though > > 2019-08-19 16:32:50 < badone> the osd read the osdmap over the wire > successfully but wrote it out to disk in a format that it could not then > read back in (unlikely) or... > > 2019-08-19 16:33:21 < badone> the map "changed" after it had been > written to disk > > 2019-08-19 16:33:46 < badone> the second is considered most likely by us > but I recognise you may not share that opinion > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

2 6

bluestore compression questions
by Andras Pataki 21 Feb '20

21 Feb '20

We're considering using bluestore compression for some of our data, and I'm not entirely sure how to interpret compression results. As an example, one of the osd perf dump results shows: "bluestore_compressed": 28089935, "bluestore_compressed_allocated": 115539968, "bluestore_compressed_original": 231079936, Am I right to interpret this as "Originally 231079936 bytes (231GB) were compressed to 28089935 bytes (28GB) and were stored in 115539968 bytes (115GB)". If so, why is there such a huge discrepancy between the compressed_allocated and compressed numbers (115GB vs 28GB)? Also, ceph df detail also shows some compression stats: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR cephfs_data_ec63_c 5 1.2 TiB 311.93k 1.7 TiB 0.02 5.4 PiB N/A N/A 311.93k 116 GiB 232 GiB Would this indicate that the compressed data takes up exactly half the space (which coincides with what the osd's are saying)? It is odd that the allocated numbers are always half the original numbers ... This is ceph Nautilus with all the defaults for compression (except turning it on for the pool in question). Any insights here would be appreciated. Thanks, Andras

2 3

pg balancer plugin unresponsive
by danjou.philippe＠yahoo.de 21 Feb '20

21 Feb '20

Hi, the pg balancer is not working at all and if I call status the plugin does not respond, it just hangs forever. Mgr restart doesnt help. I have a PG distribution issue now, how to fix this? v 14.2.5 kind regards

1 0

Module 'telemetry' has experienced an error
by alexander.v.litvak＠gmail.com 21 Feb '20

21 Feb '20

This evening I was awakened by an error message cluster: id: 9b4468b7-5bf2-4964-8aec-4b2f4bee87ad health: HEALTH_ERR Module 'telemetry' has failed: ('Connection aborted.', error(101, 'Network is unreachable')) services: I have not seen any other problems with anything else on the cluster. I disabled and enabled the telemetry module and health returned to OK status. Any ideas on what could cause the issue? As far as I understand, telemetry is a module that sends messages to an external ceph server outside of the network. Thank you for any advice,

3 2

Question about min_size for replicated and EC-pools
by Rainer Krienke 21 Feb '20

21 Feb '20

Hello, I run a ceph nautilus cluster where we use rbd for data storage and retrieval. THe rbd images metadata are in a replicated pool with 1+2 copies. The data are placed in a pool that uses erasure coding with a 4+2 profile. Now I am unsure about the exact meaing or min_size for both pools. The metadata pool has a min_size of 2, the erasure coded pool has a min_size of 5. The clusters redundancy is based on hosts. As far as I understand "min_size", this would mean that the cluster would be able to access data in the pools if one host would die. In case two hosts would be offline at the same time the metadata-pool would stop delivering data, the ec-Pool would basically be able to deliver data since min_size is irrelevant to ec-pools instead the pools K-value (==4) determines the minimal number of available OSDs to continue working. In the end I would no longer be able to read data in this situation because there is only one working OSD remaining for the replicated pool and min_size is 2. If this is correct, than does it make sense to increase the replication from 3 to 4 for the metadata pool so that I would be able to read and write data even if two hosts should go down at the same time? Is my understanding correct? Thank you very much for your help Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html

1 0

pg balancer plugin unresponsive
by Philippe D'Anjou 21 Feb '20

21 Feb '20

Hi,the pg balancer is not working at all and if I call status the plugin does not respond, it just hangs forever. Mgr restart doesnt help.I have a PG distribution issue now, how to fix this?v 14.2.5 kind regards

1 0

Re: cephfs slow, howto investigate and tune mds configuration?
by Patrick Donnelly 20 Feb '20

20 Feb '20

Hello Marc, On Tue, Feb 11, 2020 at 1:41 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Thanks Samy I will give this a try. > > It would be helpful if there is some value that shows cache misses or > so, so you have a more precise idea with how much you need to increase > the cache. I have now added a couple of GB's see if it is being used and > helps speed up things. The perf dump for the MDS includes an "ino+", "ino", and "ino- " (same for "dn") which tells you how many inodes have been added to the cache ("+"), in cache (""), and removed ("-"). You can tell if you're getting a lot of cache misses if you're seeing a lot of growth in ino+ and ino-. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Lost all Monitors in Nautilus Upgrade, best way forward?
by Sean Matheny 20 Feb '20

20 Feb '20

Hi folks, Our entire cluster is down at the moment. We started upgrading from 12.2.13 to 14.2.7 with the monitors. The first monitor we upgraded crashed. We reverted to luminous on this one and tried another, and it was fine. We upgraded the rest, and they all worked. Then we upgraded the first one again, and after it became the leader, it died. Then the second one became the leader, and it died. Then the third became the leader, and it died, leaving mon 4 and 5 unable to form a quorum. We tried creating a single monitor cluster by editing the monmap of mon05, and it died in the same way, just without the paxos negotiation first. We have tried to revert to a luminous (12.2.12) monitor backup taken a few hours before the crash. The mon daemon will start, but is flooded with blocked requests and unknown pgs after a while. For better or worse we removed the “noout” flag and 144 of 232 OSDs are now showing as down: cluster: id: ababdd7f-1040-431b-962c-c45bea5424aa health: HEALTH_ERR noout,nobackfill,norecover flag(s) set 101 osds down 9 hosts (143 osds) down 1 auth entities have invalid capabilities Long heartbeat ping times on back interface seen, longest is 15424.178 msec Long heartbeat ping times on front interface seen, longest is 14763.145 msec Reduced data availability: 521 pgs inactive, 48 pgs stale 274 slow requests are blocked > 32 sec 88 stuck requests are blocked > 4096 sec 1303 slow ops, oldest one blocked for 174 sec, mon.ntr-mon01 has slow ops too many PGs per OSD (299 > max 250) services: mon: 1 daemons, quorum ntr-mon01 (age 3m) mgr: ntr-mon01(active, since 30m) mds: cephfs:1 {0=akld2e18u42=up:active(laggy or crashed)} osd: 223 osds: 66 up, 167 in flags noout,nobackfill,norecover rgw: 2 daemons active (ntr-rgw01, ntr-rgw02) data: pools: 14 pools, 15220 pgs objects: 35.26M objects, 134 TiB usage: 379 TiB used, 1014 TiB / 1.4 PiB avail pgs: 3.423% pgs unknown 14651 active+clean 521 unknown 48 stale+active+clean io: client: 20 KiB/s rd, 439 KiB/s wr, 7 op/s rd, 54 op/s wr These luminous OSD daemons are not down, but are all in fact running. They just have no comms with the monitor: 2020-02-19 10:12:33.565680 7ff222e24700 1 osd.0 pg_epoch: 305104 pg[100.37as3( v 129516'2 (0'0,129516'2] local-lis/les=297268/297269 n=0 ec=129502/129502 lis/c 297268/297268 les/c/f 297269/297358/0 297268/297268/161526) [41,192,216,0,160,117]p41(0) r=3 lpr=305101 crt=129516'2 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray 2020-02-19 10:12:33.565861 7ff222e24700 1 osd.0 pg_epoch: 305104 pg[4.53c( v 305046'1933429 (304777'1931907,305046'1933429] local-lis/les=298009/298010 n=7350 ec=768/768 lis/c 298009/298009 les/c/f 298010/298010/0 297268/298009/298009) [0,61,103] r=0 lpr=305101 crt=305046'1933429 lcod 0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-02-19 10:12:33.566742 7ff222e24700 1 osd.0 pg_epoch: 305104 pg[100.des4( v 129516'1 (0'0,129516'1] local-lis/les=292010/292011 n=1 ec=129502/129502 lis/c 292010/292010 les/c/f 292011/292417/0 292010/292010/280955) [149,62,209,187,0,98]p149(0) r=4 lpr=305072 crt=129516'1 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray 2020-02-19 10:12:33.566896 7ff23ccd9e00 0 osd.0 305104 done with init, starting boot process 2020-02-19 10:12:33.566956 7ff23ccd9e00 1 osd.0 305104 start_boot One oddity in our deployment is that there was a test mds instance, and it is running mimic. I shut it down, as the monitor trace has an MDS call in it, but the nautilus monitors still die the same way. "mds": { "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)": 1 }, ... -11> 2020-02-18 09:50:00.800 7fd164a1a700 5 mon.ntr-mon02(a)1(leader).paxos(paxos recovering c 85448935..85449502) is_readable = 0 - now=2020-02-18 09:50:00.804429 lease_expire=0.000000 has v0 lc 85449502 -10> 2020-02-18 09:50:00.800 7fd164a1a700 5 mon.ntr-mon02(a)1(leader).paxos(paxos recovering c 85448935..85449502) is_readable = 0 - now=2020-02-18 09:50:00.804446 lease_expire=0.000000 has v0 lc 85449502 -9> 2020-02-18 09:50:00.800 7fd164a1a700 5 mon.ntr-mon02(a)1(leader).paxos(paxos recovering c 85448935..85449502) is_readable = 0 - now=2020-02-18 09:50:00.804460 lease_expire=0.000000 has v0 lc 85449502 -8> 2020-02-18 09:50:00.800 7fd164a1a700 4 set_mon_vals no callback set -7> 2020-02-18 09:50:00.800 7fd164a1a700 4 mgrc handle_mgr_map Got map version 2301191 -6> 2020-02-18 09:50:00.804 7fd164a1a700 4 mgrc handle_mgr_map Active mgr is now v1:10.31.88.17:6801/2924412 -5> 2020-02-18 09:50:00.804 7fd164a1a700 0 log_channel(cluster) log [DBG] : monmap e25: 5 mons at {ntr-mon01=v1:10.31.88.14:6789/0,ntr-mon02=v1:10.31.88.15:6789/0,ntr-mon03=v1:10.31.88.16:6789/0,ntr-mon04=v1:10.31.88.17:6789/0,ntr-mon05=v1:10.31.88.18:6789/0} -4> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client _send_to_mon log to self -3> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1 -2> 2020-02-18 09:50:00.804 7fd164a1a700 10 log_client will send 2020-02-18 09:50:00.806845 mon.ntr-mon02 (mon.1) 3 : cluster [DBG] monmap e25: 5 mons at {ntr-mon01=v1:10.31.88.14:6789/0,ntr-mon02=v1:10.31.88.15:6789/0,ntr-mon03=v1:10.31.88.16:6789/0,ntr-mon04=v1:10.31.88.17:6789/0,ntr-mon05=v1:10.31.88.18:6789/0} -1> 2020-02-18 09:50:00.804 7fd164a1a700 5 mon.ntr-mon02(a)1(leader).paxos(paxos active c 85448935..85449502) is_readable = 1 - now=2020-02-18 09:50:00.806920 lease_expire=2020-02-18 09:50:05.804479 has v0 lc 85449502 0> 2020-02-18 09:50:00.812 7fd164a1a700 -1 *** Caught signal (Aborted) ** in thread 7fd164a1a700 thread_name:ms_dispatch ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (()+0x11390) [0x7fd171e98390] 2: (gsignal()+0x38) [0x7fd1715e5428] 3: (abort()+0x16a) [0x7fd1715e702a] 4: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7fd173673bf5] 5: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7fd173667bd6] 6: (()+0x8b6c21) [0x7fd173667c21] 7: (()+0x8c2e34) [0x7fd173673e34] 8: (std::__throw_out_of_range(char const*)+0x3f) [0x7fd17367f55f] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x79ae00] 10: (MDSMonitor::tick()+0xc9) [0x79c669] 11: (MDSMonitor::on_active()+0x28) [0x785e88] 12: (PaxosService::_active()+0xdd) [0x6d4b2d] 13: (Context::complete(int)+0x9) [0x600789] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x6299a8] 15: (Paxos::finish_round()+0x76) [0x6cb276] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xbff) [0x6cc47f] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x24b) [0x6ccf2b] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x15c5) [0x5fa6f5] 19: (Monitor::_ms_dispatch(Message*)+0x4d2) [0x5fad42] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x62b046] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x6270b6] 22: (DispatchQueue::entry()+0x1219) [0x7fd1732b7e59] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fd17336836d] 24: (()+0x76ba) [0x7fd171e8e6ba] 25: (clone()+0x6d) [0x7fd1716b741d] ... Ceph versions output { "mon": { "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1, "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)": 4 }, "mgr": { "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 1, "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 1, "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)": 2 }, "osd": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 175, "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 32, "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 16 }, "mds": { "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)": 1 }, "rgw": { "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 2 }, "overall": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 175, "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 35, "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 18, "ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)": 1, "ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)": 6 } } We’ve filed a bug report with the actions of the actual cascading crash described above (when we upgraded mon01 and it became the leader): https://tracker.ceph.com/issues/44185 (parts here copied from that report) Right now we’re not sure what the best path to some sort of recovery would be. All OSD Daemons are still on Luminous, so AFAICT, we could build the monitor db from the OSDs with https://github.com/ceph/ceph/blob/luminous/doc/rados/troubleshooting/troubl… which describes using this script: #!/bin/bash hosts="ntr-sto01 ntr-sto02" ms=/tmp/mon-store/ mkdir $ms # collect the cluster map from OSDs for host in $hosts; do echo $host rsync -avz $ms root@$host:$ms rm -rf $ms ssh root@$host <<EOF for osd in /var/lib/ceph/osd/ceph-*; do ceph-objectstore-tool --data-path \$osd --op update-mon-db --mon-store-path $ms done EOF rsync -avz root@$host:$ms $ms done If this is our best idea to try, should we try the mon store from the above script on a luminous or nautilus mon daemon? Any other ideas to try at this dark hour? : \ Cheers, Sean

3 7

ceph nvme 2x replication
by Frank R 19 Feb '20

19 Feb '20

Hi all, I have noticed that RedHat is willing to support 2x replication with NVME drives. Additionally, I have seen CERN presentation where they use a 2x replication with NVME for a hyperconverged/HPC/CephFS solution. I would like to hear some opinions on whether this is really a good idea for production. Is this setup (NVME/2x replication) really only meant to be used for data that is backed up and/or can be lost without causing a catastrophe. Thanks, Frank

5 6

slow using ISCSI - Help-me
by Gesiel Galvão Bernardes 19 Feb '20

19 Feb '20

Hi, I am having an unusual slowdown using VMware with ISCSI gws. I have two ISCSI gateways with two RBD images. I have checked the following in the logs: Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock acquisition operation is already in process.2019-12-24 09:00:26.973 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:28.990 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Can anyone help-me please? Gesiel

2 14

Pool on limited number of OSDs
by Jacek Suchenia 19 Feb '20

19 Feb '20

Hello I have a cluster, (Nautilus 14.2.4) where one pool I'd like to keep on a dedicated OSDs. So I setup a rule that covers *3* dedicated OSDs (using device classes) and assigned it to pool with replication factor *3*. Only 10% PGs were assigned and rebalanced, where rest of them stuck in *undersized* state. What mechanism prevents CRUSH algorithm to assign the same set of OSDs to all PGs in a pool? How can I control it? Jacek -- Jacek Suchenia jacek.suchenia(a)gmail.com

3 5

Performance of old vs new hw?
by jesper＠krogh.cc 19 Feb '20

19 Feb '20

Hi We have some oldish servers with ssds - all on 25gbit nics. R815 AMD - 2,4ghz+ Is there significant performance benefits in moving to a new NVMe based, new cpus? +20% IOPs? + 50% IOPs? Jesper Sent from myMail for iOS

3 2

EC Pools w/ RBD - IOPs
by Anthony Brandelli (abrandel) 18 Feb '20

18 Feb '20

Hi Ceph Community, Wondering what experiences good/bad you have with EC pools for iops intensive workloads (IE: 4Kish random IO from things like VMWare ESXi). I realize that EC pools are a tradeoff between more usable capacity, and having larger latency/lower iops, but in my testing the tradeoff for small IO seems to be much worse than I had anticipated. On an all flash 3x replicated pool we’re seeing 45k random read, and 35k random write iops testing with fio on a client living on an iSCSI LUN presented to an ESXi host. Average latencies for these ops are 4.2ms, and 5.5ms, which is respectable at an io depth of 32. Take this same setup with an EC pool (k=2, m=1, tested with both ISA and jerasure, ISA does give better performance for our use case) and we see 30k random read, and 16k random write iops. Random reads see 6.5ms average, while random writes suffer with 12ms average. Are others using EC pools seeing similar hits to random writes with small IOs? Any way to improve this? Thanks, Anthony

4 5

ceph status reports: slow ops - this is related to long running process /usr/bin/ceph-osd
by Thomas 18 Feb '20

18 Feb '20

Hi, ceph status reports: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available 4 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 59 pgs inactive, 16 pgs peering Degraded data redundancy: 597/153910758 objects degraded (0.000%), 2 pgs degraded, 1 pg undersized Degraded data redundancy (low space): 23 pgs backfill_toofull 1 pgs not deep-scrubbed in time 4 pgs not scrubbed in time 3 pools have too many placement groups 164 slow requests are blocked > 32 sec 1082 stuck requests are blocked > 4096 sec 1490 slow ops, oldest one blocked for 19711 sec, daemons [osd,0,osd,175,osd,186,osd,5,osd,6,osd,63,osd,68,osd,9,mon,ld5505,mon,ld5506]... have slow ops. services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 5h) mgr: ld5507(active, since 5h), standbys: ld5506, ld5505 mds: pve_cephfs:0/1, 1 failed osd: 419 osds: 416 up, 416 in; 6024 remapped pgs data: pools: 6 pools, 8864 pgs objects: 51.30M objects, 196 TiB usage: 594 TiB used, 907 TiB / 1.5 PiB avail pgs: 0.666% pgs not active 597/153910758 objects degraded (0.000%) 52964415/153910758 objects misplaced (34.412%) 5954 active+remapped+backfill_wait 2786 active+clean 40 active+remapped+backfilling 35 activating 23 active+remapped+backfill_wait+backfill_toofull 16 peering 7 activating+remapped 1 activating+undersized+degraded 1 active+clean+scrubbing 1 active+recovering+degraded io: client: 3.5 KiB/s wr, 0 op/s rd, 0 op/s wr recovery: 551 MiB/s, 137 objects/s I'm concerned about the slow ops on osd.0 and osd.9. On the relevant OSD node I can see 2 relevant services running for hours: ceph 14795 1 99 09:58 ? 08:49:22 /usr/bin/ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph ceph 15394 1 99 09:58 ? 07:10:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph In the relevant osd log I can find similar messages: root@ld5505:~# tail -f /var/log/ceph/ceph-osd.0.log 2019-10-08 15:35:32.830 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:33.806 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:34.842 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:35.862 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) root@ld5505:~# tail -f /var/log/ceph/ceph-osd.9.log 2019-10-08 15:35:38.822 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:39.854 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:40.850 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:41.862 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) Question: How can I analyse and solve the issue with slow ops? THX

3 2

Re: 转发: Causual survey on the successful usage of CephFS on production
by Marc Roos 17 Feb '20

17 Feb '20

cephfs is considered to be stable. I am using it with only one mds for 2-3 years in low load environment without any serious issues. -----Original Message----- Sent: 17 February 2020 16:07 To: ceph-users Subject: [ceph-users] =?eucgb2312_cn?q?=D7=AA=B7=A2=3A_Causual_survey_on_the_successful_usage_ of_CephFS_on_production?= Dear Folks， I am planning a files systems project and CephFS is under serious consideration. But browsering on the web i found some negative comments on CephFS about stability and losing data. So i would like to learn more on the latest development about CephFS statbility, either a successful story or a failure Could you please speak out, if you have recently successfully deployed CephFS on production, and which version? any statbility issues with CephFS? thanks a lot, samuel huxiaoyu(a)horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

转发: Causual survey on the successful usage of CephFS on production
by huxiaoyu＠horebdata.cn 17 Feb '20

17 Feb '20

Dear Folks， I am planning a files systems project and CephFS is under serious consideration. But browsering on the web i found some negative comments on CephFS about stability and losing data. So i would like to learn more on the latest development about CephFS statbility, either a successful story or a failure Could you please speak out, if you have recently successfully deployed CephFS on production, and which version? any statbility issues with CephFS? thanks a lot, samuel huxiaoyu(a)horebdata.cn

1 0

cephfs metadata
by Frank R 17 Feb '20

17 Feb '20

Hi all, Is there a way to estimate how much storage space is required for CephFS metadata given an expected number of files in the filesystem? thx Frank

1 0

Re: centos7 / nautilus where to get kernel 5.5 from?
by Ilya Dryomov 17 Feb '20

17 Feb '20

On Fri, Feb 14, 2020 at 3:19 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I have default centos7 setup with nautilus. I have been asked to install > 5.5 to check a 'bug'. Where should I get this from? I read that the > elrepo kernel is not compiled like rhel. Hi Marc, I'm not sure what you mean by "not compiled like RHEL". Follow [1] to enable elrepo repository and then: $ sudo yum --enablerepo=elrepo-kernel install kernel-ml "ml" stands for "mainline". This is the most recent mainline stable kernel from kernel.org, it should get you 5.5.3. [1] http://elrepo.org/tiki/tiki-index.php Thanks, Ilya

2 1

Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
by Peter Woodman 17 Feb '20

17 Feb '20

Hey, I've been running a ceph cluster of arm64 SOCs on Luminous for the past year or so, with no major problems. I recently upgraded to 14.2.7, and the stability of the cluster immediately suffered. Seemed like any mon activity was subject to long pauses, and the cluster would hang frequently. Looking at ceph -s, it appeared the cluster was electing new masters very frequently - masters didn't seem to last longer than about 1-2 minutes. Looking further at the mons, two out of three of which are running on relatively slow-performing SD card storage on these SoCs, I saw them absolutely maxing out the root device's IO with writes. Logs show rocksdb constantly running compactions. I temporarily moved these mons to devices with better performing IO (but shouldn't be mons as they're also cephfs clients) and saw a sustained write rate of ~50MB/s. This seems pretty excessive, and is at least an order of magnitude higher than anything I saw when running Luminous. Not to mention this isn't really nice for SSD lifespan. As downgrading is not an option here, is there anything I can look to to figure out what exactly the mons are doing and how to prevent such heavy load? I seem to remember some bug related to telemetry, but can't find it on this list..

4 3

Erasure Profile Pool caps at pg_num 1024
by Gunnar Bandelow 17 Feb '20

17 Feb '20

Hello Everyone, i've run into problems with placement groups. We have a 12 Host Ceph-Cluster with 408 OSDs (hdd and ssd). I create a replicated pool with a large pg_num (16384). No problems everything works. If I do this with an erasure pool, i get a warning which is fixable by extending the max_pgs_per_osd and afterwards i get a health_warn in the ceph status due too few PGs. Checking the pg_num after pool creation, it is capped at 1024. I'm stuck at this point. Maybe i did something fundamentally wrong? To illustrate my steps I tried to summarize everything in a small example: # ceph -v ceph version 14.2.7 (fb8e34a687d76cd3bd45c2a0fb445432ab69b4ff) nautilus (stable) # ceph osd erasure-code-profile get myerasurehdd crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=7 m=5 plugin=jerasure technique=reed_sol_van w=8 # ceph osd crush rule dump sas_rule { "rule_id": 0, "rule_name": "sas_rule", "ruleset": 0, "type": 3, "min_size": 1, "max_size": 12, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] } # ceph osd pool create sas-pool 16384 16384 erasure myerasurehdd sas_rule Error ERANGE: pg_num 16384 size 12 would mean 196704 total pgs, which exceeds max 102000 (mon_max_pg_per_osd 250 * num_in_osds 408) # ceph tell mon.\* injectargs '--mon-max-pg-per-osd=500' mon.ceph-fs01: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) mon.ceph-fs05: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) mon.ceph-fs09: injectargs:mon_max_pg_per_osd = '500' (not observed, change may require restart) ceph-fs01:/opt/ceph-setup# ceph osd pool create sas-pool 16384 16384 erasure myerasurehdd sas_rule pool 'sas-pool' created # ceph -s cluster: id: b9471b57-95a2-4e58-8f69-b5e6048bea7c health: HEALTH_WARN Reduced data availability: 1024 pgs incomplete too few PGs per OSD (7 < min 30) # ceph osd pool get sas-pool pg_num pg_num: 1024 Best regards, Gunnar

3 2

Re: Bucket rename with
by Janne Johansson 17 Feb '20

17 Feb '20

Den fre 14 feb. 2020 kl 23:02 skrev EDH - Manuel Rios < mriosfer(a)easydatahost.com>: > Honestly, not having a function to rename bucket from admin rgw-admin is > like not having a function to copy or move. It is something basic, since if > not the workarround, it is to create a new bucket and move all the files > with the consequent loss of time and cost of computation. In addition to > the interruption. > Im sure that im not the only one administrator of rgw that need rename > some buckets, because by default system let Users for example use CAPITIAL > letter and that's not compliance. > I wonder if the "bucketnames might also become hostnames" weighed into the reluctance for renames, but I have also wanted to have renames, in the reverse situation where I did copy contents to another bucket (placed on a better place) and wanted to "copy OLD to NEW, rename OLD to OLD2, rename NEW to OLD, slowly delete OLD2" to have the shortest possible downtime while still doing a full object copy. -- May the most significant bit of your life be positive.

1 0

Extended security attributes on cephfs (nautilus) not working with kernel 5.3
by Stolte, Felix 17 Feb '20

17 Feb '20

Hi guys, I am exporting cephfs with samba using the vfs acl_xattr which stores ntfs acls in the security extended attributes. This works fine using cephfs kernel mount wither kernel version 4.15. Using kernel 5.3 I cannot access the security.ntacl attributes anymore. Attributes in user or ceph namespace is still working. Can someone enlighten me on this issue (or even better tell me how to fix it)? Best regards Felix ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

2 2

RBD-mirror instabilities
by Oliver Freyermuth 15 Feb '20

15 Feb '20

Dear Cephalopodians, running 13.2.6 on the source cluster and 14.2.5 on the rbd mirror nodes and the target cluster, I observe regular failures of rbd-mirror processes. With failures, I mean that traffic stops, but the daemons are still listed as active rbd-mirror daemons in "ceph -s", and the daemons are still running. This comes in sync with a hefty load of below messages in the mirror logs. This happens "sometimes" when some OSDs go down and up in the target cluster (which happens each night since the disks in that cluster shortly go offline during "online" smart self-tests - that's a problem in itself, but it's a cluster built from hardware that would have been trashed otherwise). The rbd daemons keep running in any case, but synchronization stops. If not all rbd mirror daemons have failed (we have three running, and it usually does not hit all of them), the "surviving" seem(s) not to take care of the images the other daemons had locked. Right now, I am eyeing with a "quick solution" of regularly restarting the rbd-mirror daemons, but if there are any good ideas on which debug info I could collect to get this analyzed and fixed, that would of course be appreciated :-). Cheers, Oliver ----------------------------------------------- 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_process_entry_safe: failed to commit journal event: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_replay_complete: replay encountered an error: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_process_entry_safe: failed to commit journal event: (125) Operation canceled 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_replay_complete: replay encountered an error: (125) Operation canceled 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dce2e05b0 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dcf47ee70 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:55.716 7f31f5b6f700 -1 rbd::mirror::ImageReplayer: 0x559dcb997680 [2/f8218221-6608-4a2b-8831-84ca0c2cb418] operator(): start failed: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0577bd16-acc4-4e9a-81f0-c698a24f8771: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=05bd4cca-a561-4a5c-ad83-9905ad5ce34e: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0e614ece-65b1-4b4a-99bd-44dd6235eb70: blacklisted detected during image replay -----------------------------------------------

2 6

Re: Bucket rename with
by Matt Benjamin 15 Feb '20

15 Feb '20

The world lived for a long time without it, but it's certainly useful. If Abhishek and/or the backport team would like this for Nautilus, I will help retarget our downstream backport (it's bigger than 22 commits with the dependencies, I believe, n.b.). Matt On Fri, Feb 14, 2020 at 5:02 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > Honestly, not having a function to rename bucket from admin rgw-admin is like not having a function to copy or move. It is something basic, since if not the workarround, it is to create a new bucket and move all the files with the consequent loss of time and cost of computation. In addition to the interruption. > > Im sure that im not the only one administrator of rgw that need rename some buckets, because by default system let Users for example use CAPITIAL letter and that's not compliance. > > KR, > Manuel > > -----Mensaje original----- > De: J. Eric Ivancich <ivancich(a)redhat.com> > Enviado el: viernes, 14 de febrero de 2020 20:47 > Para: EDH - Manuel Rios <mriosfer(a)easydatahost.com> > CC: ceph-users(a)ceph.io > Asunto: Re: [ceph-users] Bucket rename with > > On 2/4/20 12:29 PM, EDH - Manuel Rios wrote: > > Hi > > > > Some Customer asked us for a normal easy problem, they want rename a bucket. > > > > Checking the Nautilus documentation looks by now its not possible, but I checked master documentation and a CLI should be accomplish this apparently. > > > > $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny > > > > Will be backported to Nautilus? Or its still just for developer/master users? > > > > https://docs.ceph.com/docs/master/man/8/radosgw-admin/ > Given both that it's a feature and the sheer size of the PR -- 22 commits and 32 files altered -- my guess is that it will not be backported to Nautilus. However I'll invite the principals to weigh in. > > Best, > > Eric > > -- > J. Eric Ivancich > he/him/his > Red Hat Storage > Ann Arbor, Michigan, USA > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

1 0

Re: Bucket rename with
by J. Eric Ivancich 14 Feb '20

14 Feb '20

On 2/4/20 12:29 PM, EDH - Manuel Rios wrote: > Hi > > Some Customer asked us for a normal easy problem, they want rename a bucket. > > Checking the Nautilus documentation looks by now its not possible, but I checked master documentation and a CLI should be accomplish this apparently. > > $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny > > Will be backported to Nautilus? Or its still just for developer/master users? > > https://docs.ceph.com/docs/master/man/8/radosgw-admin/ Given both that it's a feature and the sheer size of the PR -- 22 commits and 32 files altered -- my guess is that it will not be backported to Nautilus. However I'll invite the principals to weigh in. Best, Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA

2 1

Very bad performance on a ceph rbd pool via iSCSI to VMware esx
by Salsa 14 Feb '20

14 Feb '20

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa

3 4

Monitor / MDS distribution over WAN
by Brian Topping 14 Feb '20

14 Feb '20

I had posted about some of this a year ago in [1] and got some really helpful answers. Fortunately, I know a lot more now and feel a lot more comfortable with the scenario. Because I didn’t understand the architecture very well, I took a pause on distributing monitors and MDS over a WAN. I want to try that now. With a hard limit on the production side of the WAN at two machines and a single monitor/MDS, it’s impossible to upgrade that machine without taking the network down. It only has a few hundred PGs, 8 OSDs and a mostly static CRUSH map. WAN latency is 4ms and there’s a 10Ge link between the production machines, so quorum will be maintained in all cases on the production side except during an upgrade. Most importantly, all the OSDs will remain on the production side of the WAN link. It seems like the worst thing that could happen under normal state is the mon/MDS on the non-prod side of the WAN may be a few clocks behind the quorum on production. In an upgrade state, one of the two production machines is taken down and quorum exists across the WAN. Performance on the cluster might be slower as a result, but everything will remain stable with a stable link. Of course, after the upgrade, quorum is returned to the production side and the normal state returns. This seems like a reasonable working model to me. Do others see holes in my logic? Thanks! Brian [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032271.html

1 0

Learning Ceph - Workshop ideas for entry level
by Ignacio Ocampo 14 Feb '20

14 Feb '20

Hi all, A group of friends and my self are documenting a hands-on workshop about Ceph https://github.com/Nafiux/ceph-workshop, for learning purposes. The idea is to provide visibility step-by-step on common scenarios, from basic usage, to disaster and recovery scenarios. We will hold a workshop next weekend, and some of the ideas for learning are: - Configure and learn how to consume Block Storage Devices - Configure and learn how to consume File Systems Storage - Configure and learn how to consume Object Storage - Simulate a disaster and recovery event by killing a node and setting up a new one - Simulate a node migration Any idea or feedback is welcome! From the way we've decided to install Ceph (ceph-ansible), the way we're configuring the cluster, suggestions on basic day-to-day operations we should learn, etc. Thanks for your support! -- Ignacio Ocampo

2 2

Announcing go-ceph v0.2.0
by John Mulligan 14 Feb '20

14 Feb '20

I'm happy to announce the very first formal release of the go-ceph API bindings. https://github.com/ceph/go-ceph/releases/tag/v0.2.0 They aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild and the ceph-csi project has plans to make use of this library. Specific questions, comments, bugs etc are best directed at our github issues tracker. PS. I was directed here to make this announcement. If this is not the appropriate venue or is considered too spammy please let me know off list. Thanks! --- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0