May 2020 - ceph-users - lists.ceph.io

Re: Kworker 100% with ceph-msgr (after upgrade to 14.2.6?)

by Ilya Dryomov

On Tue, Jan 14, 2020 at 10:31 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I think this is new since I upgraded to 14.2.6. kworker/7:3 100% > > [@~]# echo l > /proc/sysrq-trigger > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192e7e>] > [<ffffffffb0192e7e>] generic_swap+0x1e/0x30 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a20 EFLAGS: 00000206 > [Tue Jan 14 10:05:08 2020] RAX: 0000000000000072 RBX: 0000000000000060 > RCX: 0000000000000072 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000006 RSI: ffffa0c788914a7a > RDI: ffffa0c788914a42 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a20 R08: 0000000000000028 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe94984224500 > R12: ffffa0c788914a40 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0c788914a18 > R15: ffffa0c788914a78 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > > > > [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn > [libceph] > [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 > task.ti: ffffa0d3040f0000 > [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192200>] > [<ffffffffb0192200>] __x86_indirect_thunk_rax+0x0/0x20 > [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a28 EFLAGS: 00000286 > [Tue Jan 14 10:05:08 2020] RAX: ffffffffb0192e60 RBX: 0000000000000010 > RCX: 0000000000000010 > [Tue Jan 14 10:05:08 2020] RDX: 0000000000000008 RSI: ffffa0d0e108b828 > RDI: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a98 R08: 0000000000000000 > R09: 0000000000000016 > [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe949a9842280 > R12: ffffa0d0e108b818 > [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0d0e108b818 > R15: ffffa0d0e108b828 > [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) > GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 > [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: > 0000000080050033 > [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 > CR4: 00000000000627e0 > [Tue Jan 14 10:05:08 2020] Call Trace: > [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] ? sort+0x1c5/0x260 > [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] > build_snap_context+0x12b/0x290 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] > rebuild_snap_realms+0x2c/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] > rebuild_snap_realms+0x4b/0x90 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] > ceph_update_snap_trace+0x3ec/0x530 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] > handle_reply+0x359/0xc60 [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > [ceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? > kernel_recvmsg+0x3a/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? > account_entity_dequeue+0xae/0xd0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? > dequeue_entity+0x11c/0x5e0 > [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? > kernel_sendmsg+0x37/0x50 > [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] > ceph_con_workfn+0xe4/0x1530 [libceph] > [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? > __schedule+0x448/0x9c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] > process_one_work+0x17f/0x440 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] > worker_thread+0x126/0x3c0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? > manage_workers.isra.26+0x2a0/0x2a0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 > [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] > ret_from_fork_nospec_begin+0x21/0x21 > [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? > insert_kthread_work+0x40/0x40 Hi Marc, It's busy processing snapshot contexts, I think it's more or less the same issue that you reported in "ceph node crashed with these errors "kernel: ceph: build_snap_context" (maybe now it is urgent?)". Did "ceph: build_snap_context 100020c9287 ffff911a9a26bd00 fail -12" errors go away? How many snapshots do you have now? Did you file a tracker ticket? It's better to have all details in one place rather than scattered across different threads on the list. Adding Jeff and Zheng, who can take a closer look. Thanks, Ilya

1 month, 2 weeks

5
6
0 0

Re: mds lost very frequently

by Stefan Kooman

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com): > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

8 months

1
1
0 0

MDS rejects clients causing hanging mountpoint on linux kernel client

by Florian Pritz

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

8 months, 2 weeks

3
6
0 0

Provide more documentation for MDS performance tuning on large file systems

by Janek Bevendorff

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

10 months, 1 week

3
13
0 0

atime with cephfs

by Oliver Freyermuth

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

10 months, 3 weeks

4
6
0 0

Nautilus: rbd image stuck unaccessible after VM restart

by islepnev＠gmail.com

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

1 year, 1 month

4
3
1 0

Choosing suitable SSD for Ceph cluster

by Hermann Himmelbauer

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

1 year, 1 month

12
18
0 0

add debian buster stable support for ceph-deploy

by Jelle de Jong

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

1 year, 1 month

6
8
0 0

kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay)

by Jake Grimmett

Dear all, After enabling "allow_standby_replay" on our cluster we are getting (lots) of identical errors on the client /var/log/messages like Apr 29 14:21:26 hal kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay) We are using the ml kernel 5.6.4-1.el7 on Scientific Linux 7.8 Cluster and client are running Ceph v14.2.9 Setting was enabled with: # ceph fs set cephfs allow_standby_replay true [root@ceph-s1 ~]# ceph mds stat cephfs:1 {0=ceph-s3=up:active} 1 up:standby-replay 2 up:standby Is this something to worry about, or should we just disable allow_standby_replay ? any advice appreciated, many thanks Jake Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

1 year, 1 month

2
2
0 0

OSDs get full with bluestore logs

by Khodayar Doustar

Hi, I have a 3 node cluster of mimic with 9 osds (3 osds on each node). I use this cluster to test integration of an application with S3 api. The problem is that after a few days all OSD starts filling up with bluestore logs and goes down and out one by one! I cannot stop the logs and I cannot find the setting to fix this leakage, this should be a leakage in logs because it's not logical to fill up all OSD with bluefs logs. This is an example of logs which is being repeated in bluestore logs: [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path /var/lib/ceph/osd/ceph-5 . . . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 15:54:29.056488 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 15:54:30.057828 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 15:54:31.059252 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0xf5f0) [0x7f40fd47f5f0] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] 4: (BlueFS::log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x3b) [0x55603a1fad3b] 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x64) [0x55603a1da764] 6: (main()+0x2f3e) [0x55603a105e3e] 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] 8: (()+0x23115f) [0x55603a1d915f] 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- . . . And this is the output of daemonperf for one of remaining (filling up!) OSDs: [root@server2 ~]# ceph daemonperf osd.5 ------bluefs------- ------------bluestore------------- ----------osd----------- jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l rop | 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 ^C[root@server2 ~]# And this is the enospc logs when trying to start OSD: [root@server2 ~]# journalctl -u -f ceph-osd(a)4.service Failed to add match 'ceph-osd(a)4.service': Invalid argument Failed to add filters: Invalid argument [root@server2 ~]# journalctl -f -u ceph-osd(a)4.service -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10] Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon osd.4. Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon osd.4. and some more data: [root@server1 ~]# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 38 GiB 35 GiB 2.6 GiB 6.83 303 POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED .rgw.root 18 N/A N/A 1.1 KiB 0 50 GiB 4 4 63 B 4 B 1.1 KiB default.rgw.meta 19 N/A N/A 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182 B 9.0 KiB default.rgw.log 20 N/A N/A 0 B 0 50 GiB 107 107 307 KiB 205 KiB 0 B default.rgw.control 21 N/A N/A 0 B 0 50 GiB 5 5 0 B 0 B 0 B default.rgw.buckets.index 22 N/A N/A 0 B 0 50 GiB 5 5 501 B 60 B 0 B default.rgw.buckets.data 23 N/A N/A 377 MiB 0.39 50 GiB 160 160 385 B 1.1 KiB 603 MiB [root@server1 ~]# [root@server1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server1 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.0 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.1 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.2 -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 0 - host server2 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.3 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.4 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 1.04 99 osd.5 -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server3 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 0.96 78 osd.6 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.7 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.8 TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 MIN/MAX VAR: 0/1.04 STDDEV: 5.58 [root@server1 ~]# I'm kind of newbie to ceph, so any help or hint would be appreciated. Did I hit a bug or something is wrong with my configuration? Thanks a lot, Khodayar

1 year, 2 months

5
5
0 0

2021

2020

2019

ceph-users May 2020