April 2020 - ceph-users - lists.ceph.io

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay)
by Jake Grimmett 27 Aug '20

27 Aug '20

Dear all, After enabling "allow_standby_replay" on our cluster we are getting (lots) of identical errors on the client /var/log/messages like Apr 29 14:21:26 hal kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay) We are using the ml kernel 5.6.4-1.el7 on Scientific Linux 7.8 Cluster and client are running Ceph v14.2.9 Setting was enabled with: # ceph fs set cephfs allow_standby_replay true [root@ceph-s1 ~]# ceph mds stat cephfs:1 {0=ceph-s3=up:active} 1 up:standby-replay 2 up:standby Is this something to worry about, or should we just disable allow_standby_replay ? any advice appreciated, many thanks Jake Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

2 2

OSDs get full with bluestore logs
by Khodayar Doustar 21 Aug '20

21 Aug '20

Hi, I have a 3 node cluster of mimic with 9 osds (3 osds on each node). I use this cluster to test integration of an application with S3 api. The problem is that after a few days all OSD starts filling up with bluestore logs and goes down and out one by one! I cannot stop the logs and I cannot find the setting to fix this leakage, this should be a leakage in logs because it's not logical to fill up all OSD with bluefs logs. This is an example of logs which is being repeated in bluestore logs: [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path /var/lib/ceph/osd/ceph-5 . . . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 15:54:29.056488 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 15:54:30.057828 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 15:54:31.059252 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0xf5f0) [0x7f40fd47f5f0] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] 4: (BlueFS::log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x3b) [0x55603a1fad3b] 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x64) [0x55603a1da764] 6: (main()+0x2f3e) [0x55603a105e3e] 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] 8: (()+0x23115f) [0x55603a1d915f] 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- . . . And this is the output of daemonperf for one of remaining (filling up!) OSDs: [root@server2 ~]# ceph daemonperf osd.5 ------bluefs------- ------------bluestore------------- ----------osd----------- jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l rop | 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 ^C[root@server2 ~]# And this is the enospc logs when trying to start OSD: [root@server2 ~]# journalctl -u -f ceph-osd(a)4.service Failed to add match 'ceph-osd(a)4.service': Invalid argument Failed to add filters: Invalid argument [root@server2 ~]# journalctl -f -u ceph-osd(a)4.service -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10] Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon osd.4. Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon osd.4. and some more data: [root@server1 ~]# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 38 GiB 35 GiB 2.6 GiB 6.83 303 POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED .rgw.root 18 N/A N/A 1.1 KiB 0 50 GiB 4 4 63 B 4 B 1.1 KiB default.rgw.meta 19 N/A N/A 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182 B 9.0 KiB default.rgw.log 20 N/A N/A 0 B 0 50 GiB 107 107 307 KiB 205 KiB 0 B default.rgw.control 21 N/A N/A 0 B 0 50 GiB 5 5 0 B 0 B 0 B default.rgw.buckets.index 22 N/A N/A 0 B 0 50 GiB 5 5 501 B 60 B 0 B default.rgw.buckets.data 23 N/A N/A 377 MiB 0.39 50 GiB 160 160 385 B 1.1 KiB 603 MiB [root@server1 ~]# [root@server1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server1 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.0 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.1 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.2 -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 0 - host server2 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.3 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.4 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 1.04 99 osd.5 -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server3 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 0.96 78 osd.6 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.7 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.8 TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 MIN/MAX VAR: 0/1.04 STDDEV: 5.58 [root@server1 ~]# I'm kind of newbie to ceph, so any help or hint would be appreciated. Did I hit a bug or something is wrong with my configuration? Thanks a lot, Khodayar

5 5

Help
by Randy Morgan 17 Aug '20

17 Aug '20

We are seeking information on configuring Ceph to work with Noobaa and NextCloud. Randy -- Randy Morgan CSR Department of Chemistry/BioChemistry Brigham Young University randym(a)chem.byu.edu

3 2

Ceph influxDB support versus Telegraf Ceph plugin?
by victorhooi＠yahoo.com 06 Aug '20

06 Aug '20

Hi, I've read that Ceph has some InfluxDB reporting capabilities inbuilt (https://docs.ceph.com/docs/master/mgr/influx/) However, Telegraf, which is the system reporting daemon for InfluxDB, also has a Ceph plugin (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph) Just curious what people's thoughts on the two are, or what they are using in production? Which is easier to deploy/maintain, have you found? Or more useful for alerting, or tracking performance gremlins? Thanks, Victor

2 2

IPv6 connectivity gone for Ceph Telemetry
by Wido den Hollander 24 Jul '20

24 Jul '20

Hi, I was just checking on a few (13) IPv6-only Ceph clusters and I noticed that they couldn't send their Telemetry data anymore: telemetry.ceph.com has address 8.43.84.137 This server used to have Dual-Stack connectivity while it was still hosted at OVH. It seemed to have moved to Red Hat, but lost IPv6 connectivity. How can we get this back? Wido

2 1

Problem with OSD::osd_op_tp thread had timed out and other connected issues
by Jan Pekař - Imatic 21 Jul '20

21 Jul '20

Hello, I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network Cluster was empty, fresh install. We filled cluster with data (small blocks) using RGW. Cluster is now used for testing so no client was using it during my admin operations mentioned below After a while (7TB of data / 40M objects uploaded) we decided, that we increase pg_num from 128 to 256 to better spread data and to speedup this operation, I've set ceph config set mgr target_max_misplaced_ratio 1 so that whole cluster rebalance as quickly as it can. I have 3 issues/questions below: 1) I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to restart with logged heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide timed out after 150 after a while OSD's were back so I continued after a while with my tests. My question - increasing number of PG with maximal target_max_misplaced_ratio was too much for that OSDs? It is not recommended to do it this way? I had no problem with this increase before, but configuration of cluster was slightly different and it was luminous version. 2) Rebuild was still slow so I increased number of backfills ceph tell osd.* injectargs "--osd-max-backfills 10" and reduced recovery sleep time ceph tell osd.* injectargs "--osd-recovery-sleep-hdd 0.01" and after few hours I noticed, that some of my OSD's were restarted during recovery, in log I can see ... |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 start_waiting_for_healthy | I observed network graph usage and network utilization was low during recovery (10Gbit was not saturated). So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that OSD is using threads and HDD timeouts are not influencing heartbeats to other OSD's and MON. It looks like it is not true. 3) After OSD was wrongly marked down I can see that cluster has object degraded. There were no degraded object before that. Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs degraded, 8 pgs undersized It means that this OSD disconnection causes data degraded? How is it possible, when no OSD was lost. Data should be on that OSD and after peering should be everything OK. With luminous I had no problem, after OSD up degraded objects where recovered/found during few seconds and cluster was healthy within seconds. Thank you very much for additional info. I can perform additional tests you recommend because cluster is used for testing purpose now. With regards Jan Pekar -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz | +420326555326 ============ --

4 7

bluestore_default_buffered_write = true
by Adam Koczarski 20 Jul '20

20 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

Poor Windows performance on ceph RBD.
by Frank Schilder 13 Jul '20

13 Jul '20

Dear all, maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a back-end store. We have a pool of spinning disks to create large low-demand data disks, mainly for backups and other cold storage. Everything is fine when using linux VMs. However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly created linux VM. If anyone has pointers what to look for, we would be very grateful. The OpenNebula installation is more or less default. The current OS and libvirt versions we use are: Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64 libvirt-client.x86_64 4.5.0-23.el7_7.1 @updates qemu-kvm-ev.x86_64 10:2.12.0-33.1.el7 @centos-qemu-ev Some benchmark results from good to worse workloads: rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 : 450MB/s rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 230MB/s rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 190MB/s rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 150MB/s rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 : 26MB/s dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB. We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to 1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s and does not recover. It is almost as if Windows translates large sequential writes to small random writes. If anyone has seen and solved this before, please let us know. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 9

ceph qos
by 展荣臻（信泰） 26 Jun '20

26 Jun '20

Hi everyone: There are two types of qos in ceph(one based on tokenbucket algorithm,another based on mclock ). Which one I can use in nautilus production environment ？Thank you

2 1

ERROR: osd init failed: (1) Operation not permitted
by Ml Ml 22 Jun '20

22 Jun '20

Hello List, first of all: Yes - i made mistakes. Now i am trying to recover :-/ I had a healthy 3 node cluster which i wanted to convert to a single one. My goal was to reinstall a fresh 3 Node cluster and start with 2 nodes. I was able to healthy turn it from a 3 Node Cluster to a 2 Node cluster. Then the problems began. I started to change size=1 and min_size=1. Health was okay until here. Then over sudden both nodes got fenced...one node refused to boot, mons where missing, etc...to make long story short, here is where i am right now: root@node03:~ # ceph -s cluster b3be313f-d0ef-42d5-80c8-6b41380a47e3 health HEALTH_WARN 53 pgs stale 53 pgs stuck stale monmap e4: 2 mons at {0=10.15.15.3:6789/0,1=10.15.15.2:6789/0} election epoch 298, quorum 0,1 1,0 osdmap e6097: 14 osds: 9 up, 9 in pgmap v93644673: 512 pgs, 1 pools, 1193 GB data, 304 kobjects 1088 GB used, 32277 GB / 33366 GB avail 459 active+clean 53 stale+active+clean root@node03:~ # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 32.56990 root default -2 25.35992 host node03 0 3.57999 osd.0 up 1.00000 1.00000 5 3.62999 osd.5 up 1.00000 1.00000 6 3.62999 osd.6 up 1.00000 1.00000 7 3.62999 osd.7 up 1.00000 1.00000 8 3.62999 osd.8 up 1.00000 1.00000 19 3.62999 osd.19 up 1.00000 1.00000 20 3.62999 osd.20 up 1.00000 1.00000 -3 7.20998 host node02 3 3.62999 osd.3 up 1.00000 1.00000 4 3.57999 osd.4 up 1.00000 1.00000 1 0 osd.1 down 0 1.00000 9 0 osd.9 down 0 1.00000 10 0 osd.10 down 0 1.00000 17 0 osd.17 down 0 1.00000 18 0 osd.18 down 0 1.00000 my main mistakes seemd to be: -------------------------------- ceph osd out osd.1 ceph auth del osd.1 systemctl stop ceph-osd@1 ceph osd rm 1 umount /var/lib/ceph/osd/ceph-1 ceph osd crush remove osd.1 As far as i can tell, ceph waits and needs data from that OSD.1 (which i removed) root@node03:~ # ceph health detail HEALTH_WARN 53 pgs stale; 53 pgs stuck stale pg 0.1a6 is stuck stale for 5086.552795, current state stale+active+clean, last acting [1] pg 0.142 is stuck stale for 5086.552784, current state stale+active+clean, last acting [1] pg 0.1e is stuck stale for 5086.552820, current state stale+active+clean, last acting [1] pg 0.e0 is stuck stale for 5086.552855, current state stale+active+clean, last acting [1] pg 0.1d is stuck stale for 5086.552822, current state stale+active+clean, last acting [1] pg 0.13c is stuck stale for 5086.552791, current state stale+active+clean, last acting [1] [...] SNIP [...] pg 0.e9 is stuck stale for 5086.552955, current state stale+active+clean, last acting [1] pg 0.87 is stuck stale for 5086.552939, current state stale+active+clean, last acting [1] When i try to start ODS.1 manually, i get: -------------------------------------------- 2020-02-10 18:48:26.107444 7f9ce31dd880 0 ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 10210 2020-02-10 18:48:26.134417 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) backend xfs (magic 0x58465342) 2020-02-10 18:48:26.184202 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is supported and appears to work 2020-02-10 18:48:26.184209 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2020-02-10 18:48:26.184526 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2020-02-10 18:48:26.184585 7f9ce31dd880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_feature: extsize is disabled by conf 2020-02-10 18:48:26.309755 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2020-02-10 18:48:26.633926 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.642185 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.664273 7f9ce31dd880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2020-02-10 18:48:26.732154 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for clients 2020-02-10 18:48:26.732163 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400 was 8705, adjusting msgr requires for mons 2020-02-10 18:48:26.732167 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for osds 2020-02-10 18:48:26.732179 7f9ce31dd880 0 osd.1 6002 load_pgs 2020-02-10 18:48:31.939810 7f9ce31dd880 0 osd.1 6002 load_pgs opened 53 pgs 2020-02-10 18:48:31.940546 7f9ce31dd880 -1 osd.1 6002 log_to_monitors {default=true} 2020-02-10 18:48:31.942471 7f9ce31dd880 1 journal close /var/lib/ceph/osd/ceph-1/journal 2020-02-10 18:48:31.969205 7f9ce31dd880 -1 ESC[0;31m ** ERROR: osd init failed: (1) Operation not permittedESC[0m Its mounted: /dev/sdg1 3.7T 127G 3.6T 4% /var/lib/ceph/osd/ceph-1 Is there any way i can get the OSD.1 back in? Thanks a lot, mario

2 1

Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
by Dan van der Ster 10 Jun '20

10 Jun '20

This means it has been applied: # ceph osd dump -f json | jq .require_osd_release "nautilus" -- dan On Mon, Feb 17, 2020 at 11:10 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > How do you check if you issued this command in the past? > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Excessive write load on mons after upgrade > from 12.2.13 -> 14.2.7 > > Hi Peter, > > could be a totally different problem but did you run the command "ceph > osd require-osd-release nautilus" after the upgrade? > We had poor performance after upgrading to nautilus and running this > command fixed it. The same was reported by others for previous updates. > Here is my original message regarding this issue: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPV… > > We did not observe the master election problem though. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

Help! ceph-mon is blocked after shutting down and ip address changed
by occj＠qq.com 04 Jun '20

04 Jun '20

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) os :CentOS Linux release 7.7.1908 (Core) single node ceph cluster with 1 mon,1mgr,1 mds,1rgw and 12osds , but only cephfs is used. ceph -s is blocked after shutting down the machine (192.168.0.104), then ip address changed to 192.168.1.6 I created the monmap with monmap tool and update the ceph.conf , hosts file and then start ceph-mon. and the ceph-mon log: ... 2019-12-11 08:57:45.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1285.14s 2019-12-11 08:57:50.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1290.14s 2019-12-11 08:57:55.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1295.14s 2019-12-11 08:58:00.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1300.14s 2019-12-11 08:58:05.172 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1305.14s 2019-12-11 08:58:10.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1310.14s 2019-12-11 08:58:15.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1315.14s 2019-12-11 08:58:20.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1320.14s 2019-12-11 08:58:25.174 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1325.14s ... I changed IP back to 192.168.0.104 yeasterday, but all the same. # cat /etc/ceph/ceph.conf [client.libvirt] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor log file = /var/log/ceph/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor [client.rgw.ceph-node1.rgw0] host = ceph-node1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-node1.rgw0.log rgw frontends = beast endpoint=192.168.1.6:8080 rgw thread pool size = 512 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.1.0/24 fsid = e384e8e6-94d5-4812-bfbb-d1b0468bdef5 mon host = [v2:192.168.1.6:3300,v1:192.168.1.6:6789] mon initial members = ceph-node1 osd crush chooseleaf type = 0 osd pool default crush rule = -1 public network = 192.168.1.0/24 [osd] osd memory target = 7870655146

2 1

Radosgw PubSub Traffic
by Dustin Guerrero 02 Jun '20

02 Jun '20

Hey all, We’ve been running some benchmarks against Ceph which we deployed using the Rook operator in Kubernetes. Everything seemed to scale linearly until a point where I see a single OSD receiving much higher CPU load than the other OSDs (nearly 100% saturation). After some investigation we noticed a ton of pubsub traffic in the strace coming from the RGW pods like so: [pid 22561] sendmsg(77, {msg_name(0)=NULL, msg_iov(3)=[{"\21\2)\0\0\0\10\0:\1\0\0\10\0\0\0\0\0\10\0\0\0\0\0\0\20\0\0-\321\211K"..., 73}, {"\200\0\0\0pubsub.user.ceph-user-wwITOk"..., 314}, {"\0\303\34[\360\314\233\2138\377\377\377\377\377\377\377\377", 17}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL|MSG_MORE <unfinished …> I’ve checked other OSDs and only a single OSD receives these messages. I suspect its creating a bottleneck. Does anyone have an idea on why these are being generated or how to stop them? The pubsub sync module doesn’t appear to be enabled, and our benchmark is doing simple gets/puts/deletes. We’re running Ceph 14.2.5 nautilus Thank you!

2 2

OSD backups and recovery
by Ludek Navratil 30 May '20

30 May '20

HI all, what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local filesystem and backup the content using Amanda. I believe that there must a better way to do this but I couldn't find it in docs. I know that one option is to setup an archive zone, but it requires an additional ceph cluster that needs to be maintained and looked after. I would rather avoid that. How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me to recover the data correctly? Many thanks,Ludek

4 5

General question CephFS or RBD
by Willi Schiegel 29 May '20

29 May '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

2 1

Ceph and iSCSI
by Bobby 29 May '20

29 May '20

Hi all, I am new to Ceph. But I have a some good understanding of iSCSI protocol. I will dive into Ceph because it looks promising. I am particularly interested in Ceph-RBD. I have a request. Can you please tell me, if any, what are the common similarities between iSCSI and Ceph. If someone has to work on a common model for iSCSI and Ceph, what would be those significant points you would suggest to someone who has some understanding of iSCSI? Looking forward to answers. Thanks in advance :-) BR

2 1

Very bad performance on a ceph rbd pool via iSCSI to VMware esx
by Salsa 29 May '20

29 May '20

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 2

Ceph and Windows - experiences or suggestions
by Lars Täuber 29 May '20

29 May '20

Hi there! I got the task to connect a Windows client to our existing ceph cluster. I'm looking for experiences or suggestions from the community. There came two possibilities to my mind: 1. iSCSI Target on RBD exported to Windows 2. NFS-Ganesha on CephFS exported to Windows Is there a third way exporting a ceph cluster to a windows machine? I have some experiences with CephFS. We have a small cluster successfully running for linux clients. I don't have experiences with RBD or iSCSI. The Windows machine will use the space for backups. The kind of data is unknown. I expect the data to be a MS-SQL dump and user data from a sharepoint system. The Windows admin does not care whether NFS or iSCSI is used. I'd be happy if some of you could share experiences. Thanks Lars

10 10

ceph with rdma can not mount with kernel
by 李亚锋 29 May '20

29 May '20

2 1

Ceph on CentOS 8?
by Jan Kasprzak 29 May '20

29 May '20

Hello, Ceph users, does anybody use Ceph on recently released CentOS 8? Apparently there are no el8 packages neither at download.ceph.com, nor in the native CentOS package tree. I am thinking about upgrading my cluster to C8 (because of other software running on it apart from Ceph). Do el7 packages simply work? Can they be rebuilt using rpmbuild --rebuild? Or is running Ceph on C8 more complicated than that? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | sir_clive> I hope you don't mind if I steal some of your ideas? laryross> As far as stealing... we call it sharing here. --from rcgroups

9 8

rbd image naming convention
by Palanisamy 29 May '20

29 May '20

Hello Team, We've integrated Ceph cluster storage with Kubernetes and provisioning volumes through rbd-provisioner. When we're creating volumes from yaml files in Kubernetes, pv > pvc > mounting to pod, In kubernetes end pvc are showing as meaningful naming convention as per yaml file defined. But in ceph cluster, rbd image name is creating with dynamic uid. During troubleshooting time, this will be tedious to find exact rbd image. Please find the provisioning logs in below pasted snippet. kubectl get pods,pv,pvc NAME READY STATUS RESTARTS AGE pod/sleepypod 1/1 Running 0 4m9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO Delete Bound default/test-dyn-pvc ceph-rbd 4m9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-dyn-pvc Bound pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO ceph-rbd 4m11s *rbd-provisioner logs* I1121 10:59:15.009012 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2" I1121 10:59:15.009092 1 controller.go:1087] provision "default/test-dyn-pvc" class "ceph-rbd": volume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" provisioned I1121 10:59:15.009138 1 controller.go:1101] provision "default/test-dyn-pvc" class "ceph-rbd": trying to save persistentvvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" I1121 10:59:15.020418 1 controller.go:1108] provision "default/test-dyn-pvc" class "ceph-rbd": persistentvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" saved I1121 10:59:15.020476 1 controller.go:1149] provision "default/test-dyn-pvc" class "ceph-rbd": succeeded I1121 10:59:15.020802 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-dyn-pvc", UID:"cd37d2d6-cecc-4a05-9736-c8d80abde7f5", APIVersion:"v1", ResourceVersion:"24545639", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 *rbd image details in Ceph cluster end* rbd -p kube ls --long NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2 1 GiB 2 is there way to setup proper naming convention for rbd image as well during kubernetes deployment itself. Kubernetes version: v1.15.5 Ceph cluster version: 14.2.2 nautilus (stable) *Best Regards,* *Palanisamy*

2 2

cephfs file layouts, empty objects in first data pool
by Håkan T Johansson 28 May '20

28 May '20

Hi, running 14.2.6, debian buster (backports). Have set up a cephfs with 3 data pools and one metadata pool: myfs_data, myfs_data_hdd, myfs_data_ssd, and myfs_metadata. The data of all files are with the use of ceph.dir.layout.pool either stored in the pools myfs_data_hdd or myfs_data_ssd. This has also been checked by dumping the ceph.file.layout.pool attributes of all files. The filesystem has 1617949 files and 36042 directories. There are however approximately as many objects in the first pool created for the cephfs, myfs_data, as there are files. They also becomes more or fewer as files are created or deleted (so cannot be some leftover from earlier exercises). Note how the USED size is reported as 0 bytes, correctly reflecting that no file data is stored in them. POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR myfs_data 0 B 1618229 0 4854687 0 0 0 2263590 129 GiB 23312479 124 GiB 0 B 0 B myfs_data_hdd 831 GiB 136309 0 408927 0 0 0 106046 200 GiB 269084 277 GiB 0 B 0 B myfs_data_ssd 43 GiB 1552412 0 4657236 0 0 0 181468 2.3 GiB 4661935 12 GiB 0 B 0 B myfs_metadata 1.2 GiB 36096 0 108288 0 0 0 4828623 82 GiB 1355102 143 GiB 0 B 0 B Is this expected? I was assuming that in this scenario, all objects, both their data and any keys would be either in the metadata pool, or the two pools where the objects are stored. Is it some additional metadata keys that are stored in the first created data pool for cephfs? This would not be so nice in case the osd selection rules for it are using worse disks than the data itself... Btw: is there any tool to see the amount of key value data size associated with a pool? 'ceph osd df' gives omap and meta for osds, but not broken down per pool. Best regards, Håkan

4 6

RGW Multi-site Issue
by Zhenshi Zhou 26 May '20

26 May '20

Hi, I am new on rgw and try deploying a mutisite cluster in order to sync data from one cluster to another. My source zone is the default zone in the default zonegroup, structure as belows: realm: big-realm | zonegroup: default / \ master zone: default secondary zone: backup *STEP*: on source cluster: 1. radosgw-admin realm create --rgw-realm=big-realm --default 2. radosgw-admin zonegroup modify --rgw-realm big-realm --rgw-zonegroup default --master --endpoints "http://172.24.29.26:7480" 3. radosgw-admin zone modify --rgw-zonegroup default --rgw-zone default --master --endpoints "http://172.24.29.26:7480" 4. radosgw-admin user create --uid=sync-user --display-name="Synchronization User" --access-key=redhat --secret=redhat --system 5. radosgw-admin zone modify --rgw-zone=default --access-key=redhat --secret=redhat 6. radosgw-admin period update --commit on destination cluster: 1. radosgw-admin realm pull --url="http://172.24.29.26:7480 <http://172.24.29.26/>" --access-key=redhat --secret=redhat --rgw-realm=big-realm 2. radosgw-admin realm default --rgw-realm=big-realm 3. radosgw-admin period pull --url="http://172.24.29.26:7480" --access-key=redhat --secret=redhat 4. radosgw-admin zonegroup default --rgw-zonegroup=default 5. radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=backup --endpoints="http://172.24.29.29:7480" --access-key=redhat --secret=redhat --default 6. radosgw-admin period update --commit commit period on secondary zone get error: 2020-04-02 14:36:04.707 7fd8ee9376c0 1 Cannot find zone id=8c75360a-c0cf-4772-b85e-ff74726396c2 (name=backup), switching to local zonegroup configuration Sending period to new master zone 5fba7cae-47f1-4c8e-9a34-1b499c9c27f8 request failed: (2202) Unknown error 2202 failed to commit period: (2202) Unknown error 2202 radosgw-admin sync status: 2020-04-02 14:37:18.330 7f27c60676c0 1 Cannot find zone id=8c75360a-c0cf-4772-b85e-ff74726396c2 (name=backup), switching to local zonegroup configuration realm fec73799-36be-4418-abb2-9804cc83d83d (big-realm) zonegroup fc61ac2f-dc1d-421b-90af-ffe9113b9935 (default) zone 8c75360a-c0cf-4772-b85e-ff74726396c2 (backup) metadata sync failed to read sync status: (2) No such file or directory data sync source: 5fba7cae-47f1-4c8e-9a34-1b499c9c27f8 (default) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source My source cluster's version is 13.2.8, and destination cluster's is 14.2.8. I tried sync data from cluster of both 13.2.8 version and got the same error. Is there any step I was wrong or the default zone cannot be synced? Thanks

3 4

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

Ceph Apply/Commit vs Read/Write Op Latency
by John Petrini 12 May '20

12 May '20

Hello, I was hoping someone could clear up the difference between these metrics. In filestore the difference between Apply and Commit Latency was pretty clear and these metrics gave a good representation of how the cluster was performing. High commit usually meant our journals were performing poorly while high apply pointed to an OSD issue. With bluestore Apply & Commit are now tied to the same metric and it's not as clear to me what that metric is. In addition new metrics such as Read and Write Op Latency have been added. I'm led to believe that these are similar to what Apply Latency used to represent but is that actually the case? If anyone who has a better understanding of this than I do can enlighten me I'd appreciate it! Thanks, John

1 1

MDS_CACHE_OVERSIZED warning
by jesper＠krogh.cc 11 May '20

11 May '20

Hi. I have a cluster that has been running for close to 2 years now - pretty much with the same setting, but over the past day I'm seeing this warning. (and the cache seem to keep growing) - Can I figure out which clients is accumulating the inodes? Ceph 12.2.8 - is it ok just to "bump" the memory to say 128GB - any negative sideeffects? jk@ceph-mon1:~$ sudo ceph health detail HEALTH_WARN 1 MDSs report oversized cache; 3 clients failing to respond to cache pressure MDS_CACHE_OVERSIZED 1 MDSs report oversized cache mdsceph-mds1(mds.0): MDS cache is too large (91GB/32GB); 34400070 inodes in use by clients, 3293 stray files Thanks - Jesper

2 1

adding block.db to OSD
by Stefan Priebe - Profihost AG 11 May '20

11 May '20

Hello, is there anything else needed beside running: ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD} bluefs-bdev-new-db --dev-target /dev/vgroup/lvdb-1 I did so some weeks ago and currently i'm seeing that all osds originally deployed with --block-db show 10-20% I/O waits while all those got converted using ceph-bluestore-tool show 80-100% I/O waits. Also is there some tuning available to use more of the SSD? The SSD (block-db) is only saturated at 0-2%. Greets, Stefan

2 21

cephfs change/migrate default data pool
by Kenneth Waegeman 08 May '20

08 May '20

Hi all, I read in some release notes it is recommended to have your default data pool replicated and use erasure coded pools as additional pools through layouts. We have still a cephfs with +-1PB usage with a EC default pool. Is there a way to change the default pool or some other kind of migration without having to recreate the FS? Thanks! Kenneth

3 4

How to apply ceph.conf changes using new tool cephadm
by Gencer W. Genç 07 May '20

07 May '20

Hi, === NOTE: I do not see my thread in ceph-list for some reason. I don't know if list received my question or not. So, sorry if this is duplicate. === I just deployed a new cluster with cephadm instead of ceph-deploy. In tyhe past, If i change ceph.conf for tweaking, i was able to copy them and apply to all servers. But i cannot find this on new cephadm tool. I did few changes on ceph.conf but ceph is unaware of those changes. How can i apply them? I've used it with docker. Thanks, Gencer.

3 3

How many MDS servers
by Robert Ruge 07 May '20

07 May '20

Quick question Ceph guru's. For a 1.1PB raw cephfs system currently storing 191TB of data and 390 million objects (mostly small Python, ML training files etc.) how many MDS servers should I be running? System is Nautilus 14.2.8. I ask because up to know I have run one MDS with one standby-replay and occasionally it blows up with large memory consumption, 60Gb+ even though I have mds_cache_memory_limit = 32G and that was 16G until recently. It of course tries to restart on another MDS node fails again and after several attempts usually comes back up. Today I increased to two active MDS's but the question is what is the optimal number for a pretty active system? The single MDS seemed to regularly run around 1400 req/s and I often get up to six clients failing to respond to cache pressure. The current setup is: ceph fs status cephfs - 71 clients ====== +------+----------------+--------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+----------------+--------+---------------+-------+-------+ | 0 | active | a | Reqs: 447 /s | 12.0M | 11.9M | | 1 | active | b | Reqs: 154 /s | 1749k | 1686k | | 1-s | standby-replay | c | Evts: 136 /s | 1440k | 1423k | | 0-s | standby-replay | d | Evts: 402 /s | 16.8k | 298 | +------+----------------+--------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 160G | 169G | | cephfs_data | data | 574T | 140T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ | w | | x | | y | | z | +-------------+ MDS version: ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) Regards. Robert Ruge Systems & Network Manager Faculty of Science, Engineering & Built Environment [cid:image001.png@01D36789.04BE09A0] Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

4 4

Cluster blacklists MDS, can't start
by Robert LeBlanc 07 May '20

07 May '20

This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I try to unblack list them, but they still say this in the logs mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance) Looking at a pg dump, it looks like the epoch is passed that. $ ceph pg map 3.756 osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115] Last time, it seemed to just recover after about an hour all by it's self. Any way to speed this up? Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 4

4.14 kernel or greater recommendation for multiple active MDS
by Robert LeBlanc 07 May '20

07 May '20

In the Nautilus manual it recommends >= 4.14 kernel for multiple active MDSes. What are the potential issues for running the 4.4 kernel with multiple MDSes? We are in the process of upgrading the clients, but at times overrun the capacity of a single MDS server. MULTIPLE ACTIVE METADATA SERVERS <https://docs.ceph.com/docs/nautilus/cephfs/kernel-features/#multiple-active…> The feature has been supported since the Luminous release. It is recommended to use Linux kernel clients >= 4.14 when there are multiple active MDS. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 6

Re: rados buckets copy
by Andrei Mikhailovsky 07 May '20

07 May '20

Hi Manuel, My replica is 2, hence about 10TB of unaccounted usage. Andrei ----- Original Message ----- > From: "EDH - Manuel Rios" <mriosfer(a)easydatahost.com> > To: "Andrei Mikhailovsky" <andrei(a)arhont.com> > Sent: Tuesday, 28 April, 2020 23:57:20 > Subject: RE: rados buckets copy > Is your replica x3? 9x3 27... plus some overhead rounded.... > > Ceph df show including replicas , bucket stats just bucket usage no replicas. > > -----Mensaje original----- > De: Andrei Mikhailovsky <andrei(a)arhont.com> > Enviado el: miércoles, 29 de abril de 2020 0:55 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] rados buckets copy > > Hello, > > I have a problem with radosgw service where the actual disk usage (ceph df shows > 28TB usage) is way more than reported by the radosgw-admin bucket stats (9TB > usage). I have tried to get to the end of the problem, but no one seems to be > able to help. As a last resort I will attempt to copy the buckets, rename them > and remove the old buckets. > > What is the best way of doing this (probably on a high level) so that the copy > process doesn't carry on the wasted space to the new buckets? > > Cheers > > Andrei > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to > ceph-users-leave(a)ceph.io

2 3

Data loss by adding 2OSD causing Long heartbeat ping times
by Frank Schilder 07 May '20

07 May '20

Dear all, Two days ago I added very few disks to a ceph cluster and run into a problem I have never seen before when doing that. The entire cluster was deployed with mimic 13.2.2 and recently upgraded to 13.2.8. This is the first time I added OSDs under 13.2.8. I had a few hosts that I needed to add 1 or 2 OSDs to and I started with one that needed 1. Procedure was as usual: ceph osd set norebalance deploy additional OSD The OSD came up and PGs started peering, so far so good. To my surprise, however, I started seeing health-warnings about slow ping times: Long heartbeat ping times on back interface seen, longest is 1171.910 msec Long heartbeat ping times on front interface seen, longest is 1180.764 msec After peering it looked like it got better and I waited it out until the messages were gone. This took a really long time, at least 5-10 minutes. I went on to the next host and deployed 2 new OSDs this time. Same as above, but with much worse consequences. Apparently, the ping times exceeded a timeout for a very short moment and an OSD was marked out for ca. 2 seconds. Now all hell broke loose. I got health errors with the dreaded "backfill_toofull", undersized PGs and a large amount of degraded objects. I don't know what is causing what, but I ended up with data loss by just adding 2 disks. We have dedicated network hardware and each of the OSD hosts has 20GBit front and 40GBit back network capacity (LACP trunking). There are currently no more than 16 disks per server. The disks were added to an SSD pool. There was no traffic nor any other exceptional load on the system. I have ganglia resource monitoring on all nodes and cannot see a single curve going up. Network, CPU utilisation, load, everything below measurement accuracy. The hosts and network are quite overpowered and dimensioned to host many more OSDs (in future expansions). I have three questions, ordered by how urgently I need an answer: 1) I need to add more disks next week and need a workaround. Will something like this help avoiding the heartbeat time-out: ceph osd set noout ceph osd set nodown ceph osd set norebalance 2) The "lost" shards of the degraded objects were obviously still on the cluster somewhere. Is there any way to force the cluster to rescan OSDs for the shards that went orphan during the incident? 3) This smells a bit like a bug that requires attention. I was probably just lucky that I only lost 1 shard per PG. Has something similar reported before? Is this fixed in 13.2.10? Is it something new? Any settings that need to be looked at? If logs need to be collected, I can do so during my next attempt. However, I cannot risk data integrity of a production cluster and, therefore, probably not run the original procedure again. Many thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 4

CephFS with active-active NFS Ganesha
by Michael Bisig 06 May '20

06 May '20

Hi all, I am trying to setup an active-active NFS Ganesha cluster (with two Ganeshas (v3.0) running in Docker containers). I could manage to get two Ganesha daemons running using the rados_cluster backend for active-active deployment. I have the grace db within the cephfs metadata pool in an own namespace which keeps track on the node status. Now, I can mount the exposed filesystem over NFS (v4.1, v4.2) with both daemons. So far so good. __ Testing high availability resulted in an unexpected behavior for that I am not sure whether it is intentional or whether it is a configuration problem. Problem: If both are running, no E or N flags are set within the grace db, as I expect. Once, one host goes down (or is taken down) ALL clients cannot read nor write to the mounted filesystem, even the clients which are not connected to dead ganesha. In the db, I see that the dead ganesha has state NE and the active has E. This state is what I expect from the Ganesha documentation. Nevertheless, I would assume that the clients connected to the active daemon are not blocked. This state is not cleaned up by itself (e.g. after the grace period). I can unlock this situation by 'lifting' the dead node with a direct db call (using ganesha-rados-grace tool). But within an active-active deployment this is not suitable. The ganesha config looks like: ------------ NFS_CORE_PARAM { Enable_NLM = false; Protocols = 4; } NFSv4 { RecoveryBackend = rados_cluster; Minor_Versions = 1,2; } RADOS_KV { pool = "cephfsmetadata"; nodeid = "a" ; namespace = "grace"; UserId = "ganesha"; Ceph_Conf = "/etc/ceph/ceph.conf"; } MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; } EXPORT { Export_ID=101; Protocols = 4; Transports = TCP; Path = PATH; Pseudo = PSEUDO_PATH; Access_Type = RW; Attr_Expiration_Time = 0; Squash = no_root_squash; FSAL { Name = CEPH; User_Id = "ganesha"; Secret_Access_Key = CEPHXKEY; } } LOG { Default_Log_Level = "FULL_DEBUG"; } ------------ Does anyone have similar problems? Or if this behavior is by purpose, can you explain to me why this is the case? Thank you in advance for your time and thoughts. Kind regards, Michael

3 2

How to apply ceph.conf changes using new tool cephadm
by Gencer W. Genç 05 May '20

05 May '20

Hi, I just deployed a new cluster with cephadm instead of ceph-deploy. In tyhe past, If i change ceph.conf for tweaking, i was able to copy them and apply to all servers. But i cannot find this on new cephadm tool. I did few changes on ceph.conf but ceph is unaware of those changes. How can i apply them? I've used it with docker. Thanks, Gencer.

3 2

Re: RGW and the orphans
by Katarzyna Myrek 05 May '20

05 May '20

Hi Eric, Would it be possible to use it with an older cluster version (like running new radosgw-admin in the container, connecting to the cluster on 14.2.X)? Kind regards / Pozdrawiam, Katarzyna Myrek czw., 16 kwi 2020 o 19:58 EDH - Manuel Rios <mriosfer(a)easydatahost.com> napisał(a): > > Hi Eric, > > > > Are there any ETA for get those script backported maybe in 14.2.10? > > > > Regards > > Manuel > > > > > > De: Eric Ivancich <ivancich(a)redhat.com> > Enviado el: jueves, 16 de abril de 2020 19:05 > Para: Katarzyna Myrek <katarzyna(a)myrek.pl>; EDH - Manuel Rios <mriosfer(a)easydatahost.com> > CC: ceph-users(a)ceph.io > Asunto: Re: [ceph-users] RGW and the orphans > > > > There is currently a PR for an “orphans list” capability. I’m currently working on the testing side to make sure it’s part of our teuthology suite. > > > > See: https://github.com/ceph/ceph/pull/34148 > > > > Eric > > > > > > On Apr 16, 2020, at 9:26 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > > > Hi > > Thanks for the quick response. > > To be honest my cluster is getting full because of that trash and I am > at the point where I have to do the removal manually ;/. > > Kind regards / Pozdrawiam, > Katarzyna Myrek > > czw., 16 kwi 2020 o 13:09 EDH - Manuel Rios > <mriosfer(a)easydatahost.com> napisał(a): > > > Hi, > > From my experience orphans find didn't work since several releases ago, and command should be re-coded or deprecated because its not running. > > Im our cases it loops over generated shards until RGW daemon crash. > > Interested into this post, in our case orphans find takes more than 24 hours into start loop over shards, but never pass the shard 0 or 1. > > CEPH RGW devs, should provide any workaround script/ new tool or something to maintain our rgw clusters. Because with the last bugs all rgw cluster got a ton of trash, wasting resources and money. > > And manual cleaning is not trivial and easy. > > Waiting for more info, > > Manuel > > > -----Mensaje original----- > De: Katarzyna Myrek <katarzyna(a)myrek.pl> > Enviado el: jueves, 16 de abril de 2020 12:38 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] RGW and the orphans > > Hi > > Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? > > I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. > Does anyone have a good method for that? > > My cluster mostly has orphans from multipart uploads. > > > Kind regards / Pozdrawiam, > Katarzyna Myrek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

4 9

Upgrade Luminous to Nautilus on a Debian system
by Herve Ballans 04 May '20

04 May '20

Hi all, I'm planning to upgrade one on my Ceph Cluster currently on Luminous 12.2.13 / Debian Stretch (updated). On this cluster, Luminous is packaged from the official Ceph repo (deb https://download.ceph.com/debian-luminous/ stretch main) I would like to upgrade it with Debian Buster and Nautilus using the croit.io repository (deb https://mirror.croit.io/debian-nautilus/ buster main) I already prepared the steps procedure but I just want to verify one step regarding the upgrade of the ceph packages. Do I have to upgrade ceph in the same time than Debian or do i have to upgrade ceph after the Debian upgrade from Stretch to Buster ? 1) In the first case : * Replace stretch by buster in /etc/apt/sources.list * Modify the ceph.list repo by croit.io one * Upgrade the entire nodes 2) In the second case (upgrade Debian then Ceph) * Replace stretch by buster in /etc/apt/sources.list * keep the /etc/apt/sources.list.d/ceph.list as it is * Upgrade and reboot the nodes * replace the ceph.list file by croit.io * upgrade the ceph packages * restarting the Ceph services (in the right order MON -> MGR -> OSD -> MDS) Thanks a lot for your advices Regards, Hervé

3 6

ceph packages
by Mazzystr 30 Apr '20

30 Apr '20

I try to install ceph octopus rpms and some dependent packages still pull in from untrusted sources Total 8.2 MB/s | 70 MB 00:08 warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-0161c520b1c9fbf7/packages/python3-cheroot-8.2.1-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY Copr repo for ceph-el8 owned by ktdreyer 2.8 kB/s | 1.0 kB 00:00 Importing GPG key 0x2A8A41EC: Userid : "ktdreyer_ceph-el8 (None) <ktdreyer# ceph-el8(a)copr.fedorahosted.org>" Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC From : https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey… Is this ok [y/N]: I'm sorry. I can't accept this or run Ceph Octopus until all dependencies come from the ceph repo or the epel repo and built/signed correctly. This isn't our first rodeo so why the change in the release process? Thanks!

1 0

ceph-ansible question
by Szabo, Istvan (Agoda) 30 Apr '20

30 Apr '20

Hi, I've tried to create ceph luminous cluster for testing porpuses with ceph-ansible on my 3 machines hyperv vms, but I've got the below error with the following with the following osd configuration: --- dummy: osd_scenario: lvm lvm_volumes: - data: osd1lv data_vg: osd1 db: journal_lv db_vg: journal wal: journal_lv wal_vg: journal - data: osd2lv data_vg: osd2 db: journal_lv db_vg: journal wal: journal_lv wal_vg: journal I have 3 disks, 2 for osd 1 for journal, I'd like to use the journal (sdb) for both (sdc sdd) device. I've also tried just simply use: lvm_volumes: - data: /dev/sdc db: /dev/sdb wal: /dev/sdb - data: /dev/sdd db: /dev/sdb wal: /dev/sdb But this one complains about partuuid not found for sdb. I've also tried to create with parted a gpt partition to have the partuuid, but that time it complains about gpt flag. My setup is a centos 7, ansible 2.6, checked out the stable-3.2 ceph-ansible. The all.yml is like this: --- dummy: ceph_origin: repository ceph_repository: community ceph_stable_release: luminous monitor_interface: eth0 public_network: 172.19.15.0/24 cluster_network: 192.168.1.0/24 Is there anybody can show me a working configuration for ceph-volume with wal and db configuration? Thank you in advance. The ansible dump is here: failed: [ceph3] (item={'data': 'osd2lv', 'data_vg': 'osd2', 'db': 'journal_lv', 'db_vg': 'journal', 'wal': 'journal_lv', 'wal_vg': 'journal'}) => {"changed": true, "cmd": ["ceph-volume", "--cluster", "ceph", "lvm", "create", "--bluestore", "--data", "osd2/osd2lv", "--block.db", "journal/journal_lv", "--block.wal", "journal/journal_lv"], "delta": "0:00:06.029411", "end": "2020-04-28 16:48:13.252098", "item": {"data": "osd2lv", "data_vg": "osd2", "db": "journal_lv", "db_vg": "journal", "wal": "journal_lv", "wal_vg": "journal"}, "msg": "non-zero return code", "rc": 1, "start": "2020-04-28 16:48:07.222687", "stderr": "Running command: /bin/ceph-authtool --gen-print-key\nRunning command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3\nRunning command: /bin/ceph-authtool --gen-print-key\nRunning command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2\nRunning command: restorecon /var/lib/ceph/osd/ceph-2\nRunning command: chown -h ceph:ceph /dev/osd2/osd2lv\nRunning command: chown -R ceph:ceph /dev/dm-4\nRunning command: ln -s /dev/osd2/osd2lv /var/lib/ceph/osd/ceph-2/block\nRunning command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap\n stderr: got monmap epoch 1\nRunning command: ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA==\n stdout: creating /var/lib/ceph/osd/ceph-2/keyring\nadded entity osd.2 auth auth(auid = 18446744073709551615 key=AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA== with 0 caps)\nRunning command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring\nRunning command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/\nRunning command: chown -h ceph:ceph /dev/journal/journal_lv\nRunning command: chown -R ceph:ceph /dev/dm-5\nRunning command: chown -h ceph:ceph /dev/journal/journal_lv\nRunning command: chown -R ceph:ceph /dev/dm-5\nRunning command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path /dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph\n stderr: 2020-04-28 16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid\n stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time 2020-04-28 16:48:12.908859\n stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]\n stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 8: (main()+0x11a5) [0x55f13d9d4975]\n stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 10: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.\n stderr: 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time 2020-04-28 16:48:12.908859\n stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]\n stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 8: (main()+0x11a5) [0x55f13d9d4975]\n stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 10: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.\n stderr: -253> 2020-04-28 16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid\n stderr: 0> 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time 2020-04-28 16:48:12.908859\n stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)\n stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f13e067ae0]\n stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]\n stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 8: (main()+0x11a5) [0x55f13d9d4975]\n stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 10: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.\n stderr: *** Caught signal (Aborted) **\n stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd\n stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (()+0xa76e51) [0x55f13e028e51]\n stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37) [0x7fb899019337]\n stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]\n stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 12: (main()+0x11a5) [0x55f13d9d4975]\n stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 14: (()+0x4c4383) [0x55f13da76383]\n stderr: 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **\n stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd\n stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (()+0xa76e51) [0x55f13e028e51]\n stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37) [0x7fb899019337]\n stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]\n stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 12: (main()+0x11a5) [0x55f13d9d4975]\n stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 14: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.\n stderr: 0> 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **\n stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd\n stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)\n stderr: 1: (()+0xa76e51) [0x55f13e028e51]\n stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]\n stderr: 3: (gsignal()+0x37) [0x7fb899019337]\n stderr: 4: (abort()+0x148) [0x7fb89901aa28]\n stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55f13e067c54]\n stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]\n stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]\n stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]\n stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]\n stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]\n stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]\n stderr: 12: (main()+0x11a5) [0x55f13d9d4975]\n stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]\n stderr: 14: (()+0x4c4383) [0x55f13da76383]\n stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.\n--> Was unable to complete a new OSD, will rollback changes\n--> OSD will be fully purged from the cluster, because the ID was generated\nRunning command: ceph osd purge osd.2 --yes-i-really-mean-it\n stderr: purged osd.2\nTraceback (most recent call last):\n File \"/sbin/ceph-volume\", line 9, in <module>\n load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()\n File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in __init__\n self.main(self.argv)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in newfunc\n return f(*a, **kw)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 149, in main\n terminal.dispatch(self.mapper, subcommand_args)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in dispatch\n instance.main()\n File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40, in main\n terminal.dispatch(self.mapper, self.argv)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in dispatch\n instance.main()\n File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 69, in main\n self.create(args)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root\n return func(*a, **kw)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 26, in create\n prepare_step.safe_prepare(args)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line 219, in safe_prepare\n self.prepare()\n File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root\n return func(*a, **kw)\n File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line 320, in prepare\n osd_fsid,\n File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line 119, in prepare_bluestore\n db=db\n File \"/usr/lib/python2.7/site-packages/ceph_volume/util/prepare.py\", line 397, in osd_mkfs_bluestore\n raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))\nRuntimeError: Command failed with exit code -6: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path /dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph", "stderr_lines": ["Running command: /bin/ceph-authtool --gen-print-key", "Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3", "Running command: /bin/ceph-authtool --gen-print-key", "Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2", "Running command: restorecon /var/lib/ceph/osd/ceph-2", "Running command: chown -h ceph:ceph /dev/osd2/osd2lv", "Running command: chown -R ceph:ceph /dev/dm-4", "Running command: ln -s /dev/osd2/osd2lv /var/lib/ceph/osd/ceph-2/block", "Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap", " stderr: got monmap epoch 1", "Running command: ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA==", " stdout: creating /var/lib/ceph/osd/ceph-2/keyring", "added entity osd.2 auth auth(auid = 18446744073709551615 key=AQDX+6de1vEIOhAA8Fy22wWp7Z0d/lnYWpgenA== with 0 caps)", "Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring", "Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/", "Running command: chown -h ceph:ceph /dev/journal/journal_lv", "Running command: chown -R ceph:ceph /dev/dm-5", "Running command: chown -h ceph:ceph /dev/journal/journal_lv", "Running command: chown -R ceph:ceph /dev/dm-5", "Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path /dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph", " stderr: 2020-04-28 16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid", " stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time 2020-04-28 16:48:12.908859", " stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 8: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 10: (()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.", " stderr: 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time 2020-04-28 16:48:12.908859", " stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 8: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 10: (()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.", " stderr: -253> 2020-04-28 16:48:11.895597 7fb89cb53ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid", " stderr: 0> 2020-04-28 16:48:12.911181 7fb89cb53ec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fb89cb53ec0 time 2020-04-28 16:48:12.908859", " stderr: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/fastbmap_allocator_impl.h: 757: FAILED assert(available >= allocated)", " stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f13e067ae0]", " stderr: 2: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 3: (BlueFS::mount()+0x271) [0x55f13dff3411]", " stderr: 4: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 5: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 6: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 8: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 9: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 10: (()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.", " stderr: *** Caught signal (Aborted) **", " stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd", " stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1: (()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]", " stderr: 3: (gsignal()+0x37) [0x7fb899019337]", " stderr: 4: (abort()+0x148) [0x7fb89901aa28]", " stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55f13e067c54]", " stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]", " stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 12: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 14: (()+0x4c4383) [0x55f13da76383]", " stderr: 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **", " stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd", " stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1: (()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]", " stderr: 3: (gsignal()+0x37) [0x7fb899019337]", " stderr: 4: (abort()+0x148) [0x7fb89901aa28]", " stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55f13e067c54]", " stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]", " stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 12: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 14: (()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.", " stderr: 0> 2020-04-28 16:48:12.914234 7fb89cb53ec0 -1 *** Caught signal (Aborted) **", " stderr: in thread 7fb89cb53ec0 thread_name:ceph-osd", " stderr: ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", " stderr: 1: (()+0xa76e51) [0x55f13e028e51]", " stderr: 2: (()+0xf5f0) [0x7fb899ff95f0]", " stderr: 3: (gsignal()+0x37) [0x7fb899019337]", " stderr: 4: (abort()+0x148) [0x7fb89901aa28]", " stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55f13e067c54]", " stderr: 6: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x674) [0x55f13e01ae54]", " stderr: 7: (BlueFS::mount()+0x271) [0x55f13dff3411]", " stderr: 8: (BlueStore::_open_db(bool)+0x1857) [0x55f13df01827]", " stderr: 9: (BlueStore::_fsck(bool, bool)+0x3c7) [0x55f13df37707]", " stderr: 10: (BlueStore::mkfs()+0xe5a) [0x55f13df0d3ba]", " stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x55f13dabc99b]", " stderr: 12: (main()+0x11a5) [0x55f13d9d4975]", " stderr: 13: (__libc_start_main()+0xf5) [0x7fb899005505]", " stderr: 14: (()+0x4c4383) [0x55f13da76383]", " stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.", "--> Was unable to complete a new OSD, will rollback changes", "--> OSD will be fully purged from the cluster, because the ID was generated", "Running command: ceph osd purge osd.2 --yes-i-really-mean-it", " stderr: purged osd.2", "Traceback (most recent call last):", " File \"/sbin/ceph-volume\", line 9, in <module>", " load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()", " File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in __init__", " self.main(self.argv)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in newfunc", " return f(*a, **kw)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 149, in main", " terminal.dispatch(self.mapper, subcommand_args)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in dispatch", " instance.main()", " File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40, in main", " terminal.dispatch(self.mapper, self.argv)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 194, in dispatch", " instance.main()", " File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 69, in main", " self.create(args)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root", " return func(*a, **kw)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py\", line 26, in create", " prepare_step.safe_prepare(args)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line 219, in safe_prepare", " self.prepare()", " File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root", " return func(*a, **kw)", " File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line 320, in prepare", " osd_fsid,", " File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py\", line 119, in prepare_bluestore", " db=db", " File \"/usr/lib/python2.7/site-packages/ceph_volume/util/prepare.py\", line 397, in osd_mkfs_bluestore", " raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))", "RuntimeError: Command failed with exit code -6: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --bluestore-block-wal-path /dev/journal/journal_lv --bluestore-block-db-path /dev/journal/journal_lv --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 842cbbe5-3b34-4ea4-95d4-b0c7cab5fce3 --setuser ceph --setgroup ceph"], "stdout": "", "stdout_lines": []} Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 5

ceph crash hangs forever and recovery stop
by Francois Legrand 30 Apr '20

30 Apr '20

Hi everybody (again), We recently had a lot of osd crashs (more than 30 osd crashed). This is now fixed, but it triggered a huge rebalancing+recovery. More or less in the same time, we noticed that the ceph crash ls (or whatever other ceph crash command) hangs forever and never returns. And finally, the recovery process stops regularly (after ~1 hour) but it can be restarted by reseting the mgr daemon (systemctl restart ceph-mgr.target on the active manager). There is nothing in the logs (the manager still works, the service is up, the dashboard is accessible but simply the recovery stops). We also tryed to reboot the managers, but it doesn't solve the problem. I guess theses two problems should be linked, but not sure. Does anybody have a clue ? Thanks. F.

2 2

Ceph MDS - busy?
by jesper＠krogh.cc 30 Apr '20

30 Apr '20

Hi. How do I find out if the MDS is "busy" - being the one limiting CephFS metadata throughput. (12.2.8). $ time find . | wc -l 1918069 real 8m43.008s user 0m2.689s sys 0m7.818s or 3.667ms per file. In the light of "potentially batching" and a network latency of ~0.20ms to the MDS - I have a feeling that this could be significantly improved. Then I additionally tried to do the same through the NFS -ganesha gateway. For reference: Same - but on "local DAS - xfs". $ time find . | wc -l 1918061 real 0m4.848s user 0m2.360s sys 0m2.816s Same but "above local DAS over NFS": $ time find . | wc -l 1918061 real 5m56.546s user 0m2.903s sys 0m34.381s jk@ceph-mon1:~$ sudo ceph fs status [sudo] password for jk: cephfs - 84 clients ====== +------+----------------+-----------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+----------------+-----------+---------------+-------+-------+ | 0 | active | ceph-mds2 | Reqs: 1369 /s | 11.3M | 11.3M | | 0-s | standby-replay | ceph-mds1 | Evts: 0 /s | 0 | 0 | +------+----------------+-----------+---------------+-------+-------+ +------------------+----------+-------+-------+ | Pool | type | used | avail | +------------------+----------+-------+-------+ | cephfs_metadata | metadata | 226M | 16.4T | | cephfs_data | data | 164T | 132T | | cephfs_data_ec42 | data | 180T | 265T | +------------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 12.2.5-45redhat1xenial (d4b9f17b56b3348566926849313084dd6efc2ca2) luminous (stable) How can we asses where the bottleneck is and what to do to speed it up?

2 1

osd crashing and rocksdb corruption
by Francois Legrand 30 Apr '20

30 Apr '20

Hi all, *** Short version *** Is there a way to repair a rocksdb from errors "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ? *** Long version *** We operate a nautilus ceph cluster (with 100 disks of 8TB in 6 servers + 4 mons/mgr + 3 mds). We recently (Monday 20) upgraded from 14.2.7 to 14.2.8. This triggered a rebalancing of some data. Two days later (Wednesday 22) we had a very short power outage. Only one of the osd servers went down (and unfortunately died). This triggered a reconstruction of the losts osds. Operations went fine until Saturday 25 where some osds in the 5 remaining servers started to crash apparently with no reasons. We tryed to restart them, but they crashed again. We ended with 18 osd down (+ 16 in the dead server so 34 osd downs out of 100). Looking at the logs we found for all the crashed osd : -237> 2020-04-25 16:32:51.835 7f1f45527a80 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2729370997 in db/181355.sst offset 18446744073709551615 size 18446744073709551615 and 2020-04-25 16:05:47.251 7fcbd1e46a80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db erroring opening db: We also noticed that the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" was present few days before the crash. We also have some osd with this error but still up. We tryed to repair with : ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-3 repair But no success (it ends with _open_db erroring opening db). Thus does somebody have an idea to fix this or at least know if it's possible to repair and correct the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ! Thanks for your help (we are desperate because we will loose datas and are fighting to save something) !!! F.

3 17

Ceph crushtool in developer mode
by Bobby 30 Apr '20

30 Apr '20

Hi Cephers, Can we use *crushtool* in developer mode? I have deployed a fake local cluster for development purpose as described by Ceph documentation here ( https://docs.ceph.com/docs/mimic/dev/dev_cluster_deployement/) Best regards Bobby !

1 0

[ceph][nautilus] rbd-target-api Configuration does not have an entry for this host
by Ignazio Cassano 30 Apr '20

30 Apr '20

Hello All, I just installed ceph iscsi target following "manual installation" at https://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli-manual-install/ When the api server starts it seems to work but in service status it reports: apr 30 07:22:51 lab-ceph-01 rbd-target-api[58740]: Processing osd blacklist entries for this node apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: No OSD blacklist entries found apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: Reading the configuration object to update local LIO configuration apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: Configuration does not have an entry for this host(lab-ceph-01) - nothing to define to LIO apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Serving Flask app "rbd-target-api" (lazy loading) apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Environment: production apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: WARNING: This is a development server. Do not use it in a production deployment. apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: Use a production WSGI server instead. apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Debug mode: on apr 30 07:22:52 lab-ceph-01 rbd-target-api[58740]: * Running on http://[::]:5000/ (Press CTRL+C to quit) My configuration contains my hostname : cat /etc/ceph/iscsi-gateway.cfg [config] # Name of the Ceph storage cluster. A suitable Ceph configuration file allowing # access to the Ceph storage cluster from the gateway node is required, if not # colocated on an OSD node. cluster_name = ceph # Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph # drectory and reference the filename here gateway_keyring = ceph.client.admin.keyring # API settings. # The API supports a number of options that allow you to tailor it to your # local environment. If you want to run the API under https, you will need to # create cert/key files that are compatible for each iSCSI gateway node, that is # not locked to a specific node. SSL cert and key files *must* be called # 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory # on *each* gateway node. With the SSL files in place, you can use 'api_secure = true' # to switch to https mode. debug = true # To support the API, the bear minimum settings are: api_secure = false #prometheus_exporter = false # Additional API configuration options are as follows, defaults shown. # api_user = admin # api_password = admin # api_port = 5001 trusted_ip_list = lab-ceph-01 When I run gwcli command it reports: [root@lab-ceph-01 ansible]# gwcli -d Adding ceph cluster 'ceph' to the UI Fetching ceph osd information Querying ceph for state information Unable to access the configuration object : REST API failure, code : 504 Traceback (most recent call last): File "/bin/gwcli", line 4, in <module> __import__('pkg_resources').run_script('ceph-iscsi==3.4', 'gwcli') File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 654, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1441, in run_script exec(script_code, namespace, namespace) File "/usr/local/lib/python3.6/site-packages/ceph_iscsi-3.4-py3.6.egg/EGG-INFO/scripts/gwcli", line 194, in <module> File "/usr/local/lib/python3.6/site-packages/ceph_iscsi-3.4-py3.6.egg/EGG-INFO/scripts/gwcli", line 105, in main File "/usr/local/lib/python3.6/site-packages/ceph_iscsi-3.4-py3.6.egg/gwcli/gateway.py", line 84, in refresh gwcli.utils.GatewayError I installed the same on different server 2 or 3 weeks ago and and it is working, Any change in the code in last weeks ? Please, anyone could help me ? Ignazio

1 0

Shutdown nautilus cluster, start stuck in peering
by Marc Roos 30 Apr '20

30 Apr '20

Shutdown nautilus cluster, start stuck in peering I followed some manual here to do before turning all nodes off: ceph osd set norecover ; ceph osd set norebalance ; ceph osd set nobackfill ; ceph osd set nodown ; ceph osd set pause And after reboot almost al pg's were stuck in peering after unsetting these. What to do now????

1 1

New ceph cluster - cephx disabled, now without access
by Tom Verhaeg 29 Apr '20

29 Apr '20

Hi, I just installed a new cluster with cephadm (https://docs.ceph.com/docs/master/cephadm/install/) all was working fine until I disabled cephx using the following commands: ceph config set global auth_cluster_required none ceph config set global auth_service_required none ceph config set global auth_client_required none Now I keep running into: [root@cph1]# ceph health [errno 13] RADOS permission denied (error connecting to the cluster) However, in my /etc/ceph/ceph.conf there is no mention of the configuration keys mentioned above. Same goes for the cluster folder in: /var/lib/ceph/d5d206f4-8a4f-11ea-9d04-ac1f6bf8b2d3/mgr.cph1.srvnl.nl.qwnssx/config How can I remove this setting now? Obviously all ceph * related commands run into the same RADOS permission denied. Thanks, Tom

1 0

manually configure radosgw
by Patrick Dowler 29 Apr '20

29 Apr '20

I am trying to manually create a radosgw instance for a small development installation. I was able to muddle through and get a working mon, mgr, and osd (x2), but the docs for radosgw are based on ceph-deploy which is not part of the octopus release. The host systems are all lxc/lxd containers with centos 7 and the http://download.ceph.com/rpm-octopus/el7/$basearch repos. The radosgw setip that I did was essentially creating a keyring following the steps in https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/ht… as far as I could, but I have to admit that even after getting mon, mgr, and a couple of osds working I still don't grok the keyring usage... is there a more conceptual doc that explains how they are used and what has to be on each host? ** start rgw0 (systemctl start ceph-radosgw@rgw0) ** The failure on rgw0 host (journalctl -f -u ceph-radosgw(a)rgw0.service) Apr 27 17:00:20 rgw0 systemd[1]: Started Ceph rados gateway. Apr 27 17:00:20 rgw0 radosgw[8668]: 2020-04-27T17:00:20.791+0000 7f330173d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] Apr 27 17:00:20 rgw0 radosgw[8668]: failed to fetch mon config (--no-mon-config to skip) Apr 27 17:00:20 rgw0 systemd[1]: ceph-radosgw(a)rgw0.service: main process exited, code=exited, status=1/FAILURE [root@cephadm ~]# ceph mon dump dumped monmap epoch 2 epoch 2 fsid f286f47e-7043-4dca-aeb0-03fda1910378 last_changed 2020-04-20T16:23:49.311252+0000 created 2020-04-20T16:18:21.389527+0000 min_mon_release 15 (octopus) 0: [v2:10.121.149.102:3300/0,v1:10.121.149.102:6789/0] mon.mon0 ** on mon0 I also see a lot of this in the logs: 2020-04-22T00:29:41.293+0000 7f35cb1da700 0 cephx server client.rgw0: handle_request failed to decode CephXAuthenticate: buffer::end_of_buffer Any help would be appreciated. -- Patrick Dowler Canadian Astronomy Data Centre Victoria, BC, Canada

4 4

Newbie Question: CRUSH and Librados Profiling
by Bobby 29 Apr '20

29 Apr '20

Hi, It is a newbie question. I would be really thankful if you can answer it please. I want to compile the Ceph source code. Because I want to profile *Librados* and *CRUSH* function stacks. Please verify if this is the right track I am following: - I have cloned the Ceph from Ceph git repository - I have installed the build code dependencies from script *install-deps.sh* - Because I would like to use the* gdb debug* client program later, the client program will depend on the librados library, so I must compile ceph in debug mode. Therefore I would modify the parameters of ceph cmake in *do_cmake.sh* script accordingly. - Then I compile *do_cmake* *- *In build I run* make - j 32* *- *To start the developer mode, I run *make vstart.* *- *In the developer mode I can write *READ* and *WRITE* tests...compile these tests and then use some profiling tool to call the compiled executable to profile the function stacks. Is this the correct way for *CPU profiling*? Please let me know if it is fine or is there something more also. Bobby !

1 1

Lock errors in iscsi gateway
by Simone Lazzaris 29 Apr '20

29 Apr '20

Hi; I've build two iscsi gateway for our (small) ceph cluster.The cluster is a nautilus installation, 4 nodes with 9x4TB each, and it's working fine. We mainly use it via s3 object storage interface, but I've deployed also some rbd block devices and a cephfs filesystem. Now I'm trying to connect it to my xenserver installation. Xenserver doesn't speak rados, so I've build the iscsi gateways. Right now they are self-hosted on the xenserver, with plan to move them into physical boxes if/when needed. The gateways are build on centos8, tcmu-runner just cloned from git (I think it's 1.5.2). I've been able to connect them to our six nodes xenserver cluster, and now I'm trying to use it. When I attempt a migration of a VM disk, on the new iscsi volume, I've got these messages on the logfile that I find very worrying: Apr 27 17:32:21 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:22 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:22 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:23 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:25 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:25 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:26 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:26 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:27 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:27 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:28 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:28 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:29 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:29 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:30 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:30 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:31 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:31 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:32 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:32 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:33 iscsi2 tcmu-runner[2344]: tcmu_notify_lock_lost:222 rbd/rbdindex0.scsidisk0: Async lock drop. Old state 1 Apr 27 17:32:33 iscsi2 tcmu-runner[2344]: alua_implicit_transition:574 rbd/ rbdindex0.scsidisk0: Starting lock acquisition operation. Apr 27 17:32:34 iscsi2 tcmu-runner[2344]: tcmu_rbd_lock:762 rbd/rbdindex0.scsidisk0: Acquired exclusive lock. Apr 27 17:32:34 iscsi2 tcmu-runner[2344]: tcmu_acquire_dev_lock:441 rbd/ rbdindex0.scsidisk0: Lock acquisition successful Apr 27 17:32:36 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. After a while the migration fails, and I keep seend the error on the logs: Apr 27 17:36:01 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:06 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:08 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:09 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:16 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:21 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:21 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:26 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:28 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:29 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Apr 27 17:36:36 iscsi2 tcmu-runner[2344]: tcmu_rbd_has_lock:516 rbd/rbdindex0.scsidisk0: Could not check lock ownership. Error: Cannot send after transport endpoint shutdown. Any hints? Is this a bug? -- *Simone Lazzaris* *Qcom S.p.A. a socio unico* simone.lazzaris(a)qcom.it[1] | www.qcom.it[2] * LinkedIn[3]* | *Facebook[4]* [5]

3 7

Problems getting ceph-iscsi to work
by Ron Gage 29 Apr '20

29 Apr '20

Hi everyone! I have been working for the past week or so trying to get ceph-iscsi to work - Octopus release. Even just getting a single node working would be a major victory in this battle but so far, victory has proven elusive. My setup: a pair of Dell Optiplex 7010 desktops, each with 16 gig of memory and 1 boot drive (USB 3) and 3 SATA drives (500 Gb SSHD drives). No RAID controllers anywhere. Yes, I know that 3 nodes is the recommended minimum number for a production system - this isn't production (this is just seeing if the darned thing will even work). I am using Centos 8.1.1911 for the OS (4.18.0 kernel) with a basic or minimal installation (no X-Window). Single Gigabit ethernet per node. I have 2 MON and 2 Mgr installed and working, and I have a total of 6 OSDs working. I created the RBD pool (named "rbd" per the published instructions), creating it initially with 256 PGs (autoscale decided that 32 was a better choice - whatever). The cluster is green and all 6 OSDs are green (up and in). All deployment is via cephadm and all containers are running via podman. Here is where things start to fall apart. I was able to find RPM packages for targetcli and python-rtslib (called python3-rtslib) but was not able to find tcmu-runner nor ceph-iscsi packages. OK, no big deal. Time to head over to the manual install guide. I was able to build tcmu-runner, install it and apparently it is running (systemctl says it is active) so that appears to be OK. The problem is getting rbd-target-gw and rbd-target-api to work. They appear to build OK and of course, I am able to get them registered with systemd. They universally fail when trying to run them (systemctl start rbd-target-gw or systemctl start rbd-target-api). Both report failure. Looking in journalctl -xe shows no hints at all regarding why they failed (only that they did). Looking in /var/log/rbd-target-api/ show nothing at all (no files). Likewise in /var/log/rbd-target-gw/ (no files). HELP!! Now, some possibly germane questions: 1) are any other Ceph services required for ceph-iscsi to work like RADOSgw? 2) since there are no apparent packages available for ceph-iscsi, can anything be inferred to the production-readiness of the subsystem? 3) are there any known errata or missing steps in the instructions for getting ceph-iscsi to work? Thanks! Ron Gage

2 3

How to debug ssh: ceph orch host add ceph01 10.10.1.1
by Ml Ml 29 Apr '20

29 Apr '20

Hello List, i did: root@ceph01:~# ceph cephadm set-ssh-config -i /tmp/ssh_conf root@ceph01:~# cat /tmp/ssh_conf Host * User root StrictHostKeyChecking no UserKnownHostsFile /dev/null root@ceph01:~# ceph config-key set mgr/cephadm/ssh_identity_key -i /root/.ssh/id_rsa set mgr/cephadm/ssh_identity_key root@ceph01:~# ceph config-key set mgr/cephadm/ssh_identity_pub -i /root/.ssh/id_rsa.pub set mgr/cephadm/ssh_identity_pub But i get: root@ceph01:~# ceph orch host add ceph01 10.10.1.1 Error ENOENT: Failed to connect to ceph01 (10.10.1.1). Check that the host is reachable and accepts connections using the cephadm SSH key root@ceph01:~# ceph config-key get mgr/cephadm/ssh_identity_key => this shows my private key How can i debug this? root@ceph01:~# ssh 10.10.1.1 or root@ceph01:~# ssh ceph01 work without a prompt or key error. I am using 15.2.0. Thanks, Michael

3 3

CDS Pacific: Dashboard planning summary
by Lenz Grimmer 29 Apr '20

29 Apr '20

Hi all, a few weeks ago, a number of virtual Ceph Developer Summit meetings took place as a replacement for the in-person summit that was planned as part of Cephalocon in Seoul: https://pad.ceph.com/p/cds-pacific The Ceph Dashboard team also participated in these and held three video conference meetings to lay out our plans for the Pacific release. For details, please take a look at our notes at this Etherpad: https://pad.ceph.com/p/ceph-dashboard-pacific-priorities We tried to identify a few "themes", outlining individual tasks which we keep track of in the tracker.ceph.com bug tracker. The tracker issues should be used for discussing and defining the tasks at hand. A key theme for the upcoming Ceph Pacific release is the intention to further deepen and enhance the integration and support with cephadm and the orchestrator. For Ceph octopus, we tried focusing on the most common day-2 operation which is OSD management, but going forward we would like to also support the deployment and management of all other Ceph-related services that can be rolled out via cephadm and the orchestrator. In a hopefully not so distant future, we would like to be able to use the dashboard as a kind of "graphical installer", that guides the user through the entire installation deployment process of a Ceph cluster from scratch (well, almost: starting from an initial Mon+Mgr deployment). Another key theme is closing feature gaps: the various services of a Ceph cluster like RBD or RGW are constantly evolving and getting new features, so we always are trying to catch up with the latest developments there. We're also looking into enhancing our monitoring/alerting support and integration with Grafana and Prometheus. Last but not least, we always try to enhance and improve existing functionality and work on better usability and user experience. This also includes bigger refactoring work or updating key components that the dashboard depends on. As always, we would like the dashboard to be an application that Ceph administrators like and actually *want* to use to perform their jobs, so we are very keen on getting your feedback here! If there is anything you are missing or if you find any part of the dashboard to be confusing or not helpful, we'd like to know about it! Please get in touch with us to share your impressions and ideas. The best way to do this is to join the #ceph-dashboard IRC channel on OFTC or by filing a bug report via the tracker: https://tracker.ceph.com/projects/mgr/issues/new Thank you, Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

Upgrading to Octopus
by Simon Sutter 29 Apr '20

29 Apr '20

Hello everybody In octopus there are some interesting looking features, so I tried to upgrading my Centos 7 test nodes, according to: https://docs.ceph.com/docs/master/releases/octopus/ Everything went fine and the cluster is healthy. To test out the new dashboard functions, I tried to install it, but there are missing dependencies: yum install ceph-mgr-dashboard.noarch ..... --> Finished Dependency Resolution Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-routes Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-jwt Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-cherrypy Installing them with pip3 does of course make no difference, because those are yum dependencies. Does anyone know a workaround? Do I have to upgrade to Centos 8 for this to work? Thanks in advance, Simon

5 12

Re: RGW and the orphans
by Katarzyna Myrek 29 Apr '20

29 Apr '20

Hi @Eric Ivancich my cluster has some history and trash gathered over the years. Most (terabytes) is from https://tracker.ceph.com/issues/43756. I was able to reproduce the problem on my LAB and it is for sure connected with https://tracker.ceph.com/issues/43756. When you are on a version older than 14.2.8 you would need to apply lifecycle policy which tries to abort interrupted multiparts older than x days. And when the bucket index is sharded then the broken/un-cancellable MPs are born. To test it I can use s3cmd. My LAB cluster was upgraded to 14.2.8 to make sure the new version does not do cleanup automagically. Here is my procedure (I truncated my personal data): *s3cmd --access_key= --secret_key= --host= --host-bucket= multipart s3://kate-mp-issue* *s3://kate-mp-issue/Initiated Path Id2020-04-06T07:48:55.323Z s3://kate-mp-issue/bottest_20200406T074855.img 2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS* *s3cmd --access_key= --secret_key= --host= --host-bucket= abortmp s3://kate-mp-issue/bottest_20200406T074855.img 2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS* *ERROR: S3 error: 404 (NoSuchUpload)* *RGW logs:* 2020-04-29 07:24:23.126 7fb21b819700 1 ====== starting new request req=0x7fb21b8128d0 ===== 2020-04-29 07:24:23.126 7fb21b819700 1 ====== req done req=0x7fb21b8128d0 op status=0 http_status=200 latency=0s ====== 2020-04-29 07:24:23.126 7fb21b819700 1 civetweb: 0x381a000: IP - - [29/Apr/2020:07:24:22 +0000] "GET /kate-mp-issue/?location HTTP/1.1" 200 275 - - 2020-04-29 07:24:23.202 7fb21b819700 1 ====== starting new request req=0x7fb21b8128d0 ===== 2020-04-29 07:24:23.202 7fb21b819700 1 ====== req done req=0x7fb21b8128d0 op status=-2009 *http_status=404* latency=0s ====== 2020-04-29 07:24:23.202 7fb21b819700 1 civetweb: 0x381a000: Ip - - [29/Apr/2020:07:24:22 +0000] "DELETE /kate-mp-issue/bottest_20200406T074855.img?uploadId=2~-9SKkHzGKXYX_zNdHNs_S8RY9hWjISS HTTP/1.1" *404* 439 - - So basically I need to remove those ghost entries from the list of interrupted multiparts and clean up objects which are left overs. As far as I understand I would need to go over every object in the pool with `rados ls`, then compare the output with `radosgw-admin bi list (done for every bucket)` and with new command `radosgw-admin radoslist` and remove objects which are on 1 but not on 2 and 3. Plus clean up the interrupted multipart list. Is that correct? @EDH - Manuel Rios <mriosfer(a)easydatahost.com> Is that your method also? I really need to clean up the terbaytes of the leftovers, because my prod cluster is getting full. And now buying anything is not an option (harsh times due to pandemy). Kind regards / Pozdrawiam, Katarzyna Myrek Kind regards / Pozdrawiam, Katarzyna Myrek wt., 28 kwi 2020 o 19:45 EDH - Manuel Rios <mriosfer(a)easydatahost.com> napisał(a): > Im prettty sure that you got the same issue than we already reported : > > https://tracker.ceph.com/issues/43756 > > Garbage and garbage stored into our OSD without be able to cleanup wasting > a lot of space. > > As you can see its solved in the new versions but... the last versión > didnt have any "scrub" or similar system to fix the garbage generated in > the past versions. > > As result , even big companies got their RGW plattform with tons of TB > wasted. > > Eric, Is there a way to ask you to develop(RGW Team) a system to clean our > rgw clusters like rgw bucket scrub? > > I talked with Cbodley and he explained how to it manually but the process > is extremely complex. > > We already calculated that at least a 25% of our rgw cluster is garbage > (100TB), and our options right now: > > - Deploy a new cluster a move rgw Users one by one with their buckets with > an external copy, hopping in the last nautilus version this not happen > again (Not usefull option and not transparent) > - Buy disk and disk waiting for a solution as External Tool (No sure to > continue this way) > - Hire external developers with knowleage of ceph and create a private > tool for that. (Developers with Ceph Core/Rgw knowleage will be no easy to > find) > > Here: ceph version 14.2.8 > > > > -----Mensaje original----- > De: Eric Ivancich <ivancich(a)redhat.com> > Enviado el: martes, 28 de abril de 2020 18:39 > Para: Katarzyna Myrek <katarzyna(a)myrek.pl> > CC: ceph-users(a)ceph.io > Asunto: [ceph-users] Re: RGW and the orphans > > Hi Katarzyna, > > Incomplete multipart uploads are not considered orphans. > > With respect to the 404s…. Which version of ceph are you running? What > tooling are you using to list and cancel? Can you provide a console > transcript of the listing and cancelling? > > Thanks, > > Eric > > -- > J. Eric Ivancich > he / him / his > Red Hat Storage > Ann Arbor, Michigan, USA > > > On Apr 28, 2020, at 2:57 AM, Katarzyna Myrek <katarzyna(a)myrek.pl> wrote: > > > > Hi all > > > > I am afraid that there is even more thrash available - running > > rgw-orphan-list does not find everything. Like I still have broken > > multiparts -> when I do s3cmd multipart I get a list of > > "pending/interrupted multiparts". When I try to cancel such multipart > > I get 404. > > > > Does anyone have a method for cleanup of such things? Or even a list > > of tasks which should be run regularly on clusters with rgw ? > > > > > > Kind regards / Pozdrawiam, > > Katarzyna Myrek > > > > > > wt., 21 kwi 2020 o 09:57 Janne Johansson <icepic.dz(a)gmail.com> > napisał(a): > >> > >> Den tis 21 apr. 2020 kl 07:29 skrev Eric Ivancich <ivancich(a)redhat.com > >: > >>> > >>> Please be certain to read the associated docs in both: > >>> > >>> doc/radosgw/orphans.rst > >>> doc/man/8/rgw-orphan-list.rst > >>> > >>> so you understand the limitations and potential pitfalls. Generally > this tool will be a precursor to a large delete job, so understanding > what’s going on is important. > >>> I look forward to your report! And please feel free to post additional > questions in this forum. > >>> > >> > >> Where are those? > >> https://github.com/ceph/ceph/tree/master/doc/man/8 > >> https://github.com/ceph/ceph/tree/master/doc/radosgw > >> don't seem to contain them in master. Nor in nautilus branch or octopus. > >> > >> This whole issue feels weird, rgw (or its users) produces dead > >> fragments of mulitparts, orphans and whatnot that needs cleaning up > sooner or later and the info we get is that the old cleaner isn't meant to > be used, it hasn't worked for a long while, there is no fixed version, > perhaps there is a script somewhere with caveats. This (slightly > frustrated) issue is of course on top of "bi trim" > >> "bilog trim" > >> "mdlog trim" > >> "usage trim" > >> > >> "datalog trim" > >> > >> "sync error trim" > >> > >> "gc process" > >> > >> "reshard stale-instances rm" > >> > >> > >> > >> that we rgw admins are supposed to know when to run, how often, what > their quirks are and so on. > >> > >> > >> 'Docs' for rgw means "datalog trim" --help says "trims the datalog", > and the long version on the web would be "this operation trims the datalog" > or something that doesn't add anything more. > >> > >> > >> > >> > >> -- > >> > >> "Grumpy cat was an optimist" > >> > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io >

1 0

rados buckets copy
by Andrei Mikhailovsky 28 Apr '20

28 Apr '20

Hello, I have a problem with radosgw service where the actual disk usage (ceph df shows 28TB usage) is way more than reported by the radosgw-admin bucket stats (9TB usage). I have tried to get to the end of the problem, but no one seems to be able to help. As a last resort I will attempt to copy the buckets, rename them and remove the old buckets. What is the best way of doing this (probably on a high level) so that the copy process doesn't carry on the wasted space to the new buckets? Cheers Andrei

1 0

Nautilus upgrade causes spike in MDS latency
by Josh Haft 28 Apr '20

28 Apr '20

Hi, I upgraded from 13.2.5 to 14.2.6 last week and am now seeing significantly higher latency on various MDS operations. For example, the 2min rate of ceph_mds_server_req_create_latency_sum / ceph_mds_server_req_create_latency_count for an 8hr window last Monday prior to the upgrade was an average of 2ms. Today, however the same stat shows 869ms. Other operations including open, readdir, rmdir, etc. are also taking significantly longer. Here's a partial example of an op from dump_ops_in_flight: { "description": "client_request(client.342513090:334359409 create #...)", "initiated_at": "2020-04-13 15:30:15.707637", "age": 0.19583208099999999, "duration": 0.19767626299999999, "type_data": { "flag_point": "submit entry: journal_and_reply", "reqid": "client.342513090:334359409", "op_type": "client_request", "client_info": { "client": "client.342513090", "tid": 334359409 }, "events": [ { "time": "2020-04-13 15:30:15.707637", "event": "initiated" }, { "time": "2020-04-13 15:30:15.707637", "event": "header_read" }, { "time": "2020-04-13 15:30:15.707638", "event": "throttled" }, { "time": "2020-04-13 15:30:15.707640", "event": "all_read" }, { "time": "2020-04-13 15:30:15.781935", "event": "dispatched" }, { "time": "2020-04-13 15:30:15.785086", "event": "acquired locks" }, { "time": "2020-04-13 15:30:15.785507", "event": "early_replied" }, { "time": "2020-04-13 15:30:15.785508", "event": "submit entry: journal_and_reply" } ] } } This along with every other 'create' op I've seen has a 50ms+ delay between all_read and dispatched events - what is happening during this time? I'm not sure what I'm looking for the in the MDS debug logs. We have a mix of clients from 12.2.x through 14.2.8; my plan was to upgrade those pre-Nautilus clients this week. There is only a single MDS rank with 1 backup. Other functions of this cluster - RBDs and RGW - do not appear impacted so this looks limited to the MDS. I did not observe this behavior after upgrading a dev cluster last month. Has anyone seen anything similar? Thanks for any assistance! Josh

3 6

How to recover files from cephfs data pool
by Edison Shadabi 28 Apr '20

28 Apr '20

Hi Ceph folks, I am relatively new to Ceph cluster and I hope I can quickly receive some help here. I would like to recover files from cephfs data pool. Someone wrote that inode linkage and file names are stored in omap data of objects in metadata pool. I cant find any information about the structure of omap data of the objects in metadata pool to help me write for example a script to retrieve filenames and the related objects So I can use for example “rados get” to retrieve those files. Are there any working script that would traverse through all the metadata pool to find out file names corresponded to objects in data pool? /Ed

1 0

Bucket sync across available DCs
by Szabo, Istvan (Agoda) 28 Apr '20

28 Apr '20

Hi, is there a way to synchronize a specific bucket by Ceph across the available datacenters? I've just found multi site setup but that one sync the complete cluster, which is equal to failover solution. For me just 1 bucket. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Bucket dynamically resharded to 65521 shards - resharding manually won't work
by gl＠productsup.com 28 Apr '20

28 Apr '20

Hello, running Ceph Nautilus 14.2.4, we encountered this documented dynamic resharding issue: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-November/037531.ht… We disabled dynamic resharding in the configuration, and attempted to reshard to 1 shard: # radosgw-admin reshard add --bucket files --num-shards 1 --yes-i-really-mean-it However, it achieved nothing, and the bucket is now stuck in resharding status. It is impossible to clear the resharding flag (I have tried the bucket check --fix operation with no avail) # radosgw-admin reshard cancel --bucket=files 2020-04-28 11:47:18.721 7fd213b969c0 -1 ERROR: failed to remove entry from reshard log, oid=reshard.0000000000 tenant= bucket=files # radosgw-admin bucket reshard --bucket files --num-shards 1 ERROR: the bucket is currently undergoing resharding and cannot be added to the reshard list at this time

1 0

is ceph balancer doing anything?
by Andrei Mikhailovsky 28 Apr '20

28 Apr '20

Hello everyone, A few weeks ago I have enabled the ceph balancer on my cluster as per the instructions here: [ https://docs.ceph.com/docs/mimic/mgr/balancer/ | https://docs.ceph.com/docs/mimic/mgr/balancer/ ] I am running ceph version: ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) The cluster has 48 osds (40 osds in hdd pools and 8 osds in ssd pool) Currently, the balancer status is showing as Active. # ceph balancer status { "active": true, "plans": [], "mode": "upmap" } The health status of the cluster is: health: HEALTH_OK Previously, I've used the old REWEIGHT to change the placement of data as I've seen very uneven usage (ranging from about 60% usage on some OSDs to over 90% on others). So, I have a number of osds with reweight of 1 an some going down to 0.75. At the moment the osd usage ranges between about 65% to to just under 90%, so still a huge variation. After switching on the balancer, I've not actually seen any activity or data migration, so I am not sure if the balancer is working at all. Could someone tell me how do I check if balancing is doing its job? The second question is as the balancer is now switched on, do I suppose to set the reweight values back to their default value of 1? Many thanks

2 1

why the export v2 image is so small compare with without it?
by linghucongsong 28 Apr '20

28 Apr '20

Hi all! I export the rbd image use the export v2 format and without it. The difference is the export v2 format image is more smaller than the image which do not use export v2 format. Can any body tell me why they are size is difference so huge? Thanks very much! root@controller:/mnt# rbd du images/35d69ca5-b4f7-499e-9719-331eee498bc4 NAME PROVISIONED USED 35d69ca5-b4f7-499e-9719-331eee498bc4@snap 40GiB 40GiB 35d69ca5-b4f7-499e-9719-331eee498bc4 40GiB 0B <TOTAL> 40GiB 40GiB root@controller:/mnt# rbd export --export-format 2 images/35d69ca5-b4f7-499e-9719-331eee498bc4 ./v2image Exporting image: 100% complete...done. root@controller:/mnt# du -sh ./v2image 9.8G ./v2image root@controller:/mnt# rbd export images/35d69ca5-b4f7-499e-9719-331eee498bc4 ./image Exporting image: 100% complete...done. root@controller:/mnt# du -sh ./image 40G ./image

1 0

HBase/HDFS on Ceph/CephFS
by jesper＠krogh.cc 27 Apr '20

27 Apr '20

Hi We have an 3 year old Hadoop cluster - up for refresh - so it is time to evaluate options. The "only" usecase is running an HBase installation which is important for us and migrating out of HBase would be a hazzle. Our Ceph usage has expanded and in general - we really like what we see. Thus - Can this be "sanely" consolidated somehow? I have seen this: https://docs.ceph.com/docs/jewel/cephfs/hadoop/ But it seem really-really bogus to me. It recommends that you set: pool 3 'hadoop1' rep size 1 min_size 1 Which would - if I understand correct - be disastrous. The Hadoop end would replicated in 3 across - but within Ceph the replication would be 1. The 1 replication in ceph means pulling the OSD node would "gaurantee" the pg's to go inactive - which could be ok - but there is nothing gauranteeing that the other Hadoop replicas are not served out of the same OSD-node/pg? In which case - rebooting an OSD node would bring the hadoop cluster unavailable. Is anyone serving HBase out of Ceph - how does the stadck and configuration look? If I went for 3 x replication in both Ceph and HDFS then it would definately work, but 9x copies of the dataset is a bit more than what looks feasible at the moment. Thanks for your reflections/input. Jesper

4 3

Ceph reporting out-of-charts metrics (Nautilus 14.2.8)
by David Bartoš 27 Apr '20

27 Apr '20

Hi, thought i might ask here since i was unable to find anything similar to my issue.Perhaps someone might have an idea. In our org we are running currently few nautilus clusters (14.2.2 , 14.2.4, 14.2.8) But strangely enough the clusters with 14.2.8 are reporting weird metrics of pool R/W. To be more specific, from time to time some random huge spikes in orders of TB/s appear in our graphs , whereas the real usage is GB/s (Clusters are monitored by Prometheus). First to blame was our exporters, so i tried the one built in ceph-dashboard , but the values of those metrics were same (+- few bytes). I went as far as writing a small program to talk directly to the ceph API, but after i processed the output, data was also the same. I can provide more details if someone has a clue what might be the cause(Graphs, logs etc...). As far as the rest of the clusters goes, their graphs does not contain any spikes as the ones of the 14.2.8 clusters. Thanks for any responses/help in advance.

1 0

Existing Cluster to cephadm - mds start failing
by Ashley Merrick 27 Apr '20

27 Apr '20

Completed the migration of an existing Ceph cluster on Octopus to cephadm. All OSD/MON/MGR moved fine, however upon running the command to setup some new MDS for cephfs they both failed to start. After looking into the cephadm log's I found the following error: Apr 13 06:26:15 sn-s01 systemd[1]: Started Ceph mds.cephfs.sn-s01.snkhfd for b1db6b36-0c4c-4bce-9cda-18834be0632d. Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+0000 7f380b908700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [1] but i only support [2] Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+0000 7f380b107700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [1] but i only support [2] Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+0000 7f380a906700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [1] but i only support [2] Apr 13 06:26:16 sn-s01 bash[3520809]: failed to fetch mon config (--no-mon-config to skip This cluster is running with cephx disabled, I imported the ceph.conf into cephadm fine and this has worked fine when the other services have started, but from the above error looks like maybe the mds is not checking if cephx is enabled or disabled before trying to communicate to the mon's?

1 2

osd crashing and rocksdb corruption
by Francois Legrand 27 Apr '20

27 Apr '20

Hi all, *** Short version *** Is there a way to repair a rocksdb from errors "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ? *** Long version *** We operate a nautilus ceph cluster (with 100 disks of 8TB in 6 servers + 4 mons/mgr + 3 mds). We recently (Monday 20) upgraded from 14.2.7 to 14.2.8. This triggered a rebalancing of some data. Two days later (Wednesday 22) we had a very short power outage. Only one of the osd servers went down (and unfortunately died). This triggered a reconstruction of the losts osds. Operations went fine until Saturday 25 where some osds in the 5 remaining servers started to crash apparently with no reasons. We tryed to restart them, but they crashed again. We ended with 18 osd down (+ 16 in the dead server so 34 osd downs out of 100). Looking at the logs we found for all the crashed osd : -237> 2020-04-25 16:32:51.835 7f1f45527a80 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2729370997 in db/181355.sst offset 18446744073709551615 size 18446744073709551615 and 2020-04-25 16:05:47.251 7fcbd1e46a80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _open_db erroring opening db: We also noticed that the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" was present few days before the crash. We also have some osd with this error but still up. We tryed to repair with : ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-3 repair But no success (it ends with _open_db erroring opening db). Thus does somebody have an idea to fix this or at least know if it's possible to repair and correct the "Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch" and "_open_db erroring opening db" ! Thanks for your help (we are desperate because we will loose datas and are fighting to save something) !!! F.

1 0

Re: HBase/HDFS on Ceph/CephFS
by Xiaoxi Chen 27 Apr '20

27 Apr '20

RBD is never a workable solution unless you want to pay the cost of double-replication in both HDFS and Ceph. I think the right approach is thinking about other implementation of the FileSystem interface, like s3a and localfs. s3a is straight forward, ceph rgw provide s3 interface and s3a is stable and well tested in Hadoop ecosystem, just run it. There are a few other in-house solution offered by some vendor that integrating librgw into the s3a driver so it saves one extra hop and the management/LB cost of maintaining an RGW cluster. local filesystem is a bit tricky, we just tried a POC that mounting CephFS into every hadoop , configure Hadoop using LocalFS with Replica = 1. Which end up with each data only write once into cephfs and cephfs take care of the data durability. There was a libcephfs-jni but it is significantly out of date and seems be abandoned, which is a pity. For both solutions for sure you lost the locality , but trading for better scalability and compute/storage separation. -Xiaoxi Marc Roos <M.Roos(a)f1-outsourcing.eu> 于2020年4月24日周五下午4:00写道： > > I think the idea behind pool size of 1, is that hadoop already writes > copies to 2 other pools(?). > > However that leaves the possibility that pg's of these 3 pools can maybe > share an osd, and if that osd fails, you loose data in these pools. I > have no idea what the chances are that the same data of different pools > can end up on the same osd. > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] HBase/HDFS on Ceph/CephFS > > Hi > > We have an 3 year old Hadoop cluster - up for refresh - so it is time to > evaluate options. The "only" usecase is running an HBase installation > which is important for us and migrating out of HBase would be a hazzle. > > Our Ceph usage has expanded and in general - we really like what we see. > > Thus - Can this be "sanely" consolidated somehow? I have seen this: > https://docs.ceph.com/docs/jewel/cephfs/hadoop/ > But it seem really-really bogus to me. > > It recommends that you set: > pool 3 'hadoop1' rep size 1 min_size 1 > > Which would - if I understand correct - be disastrous. The Hadoop end > would replicated in 3 across - but within Ceph the replication would be > 1. > The 1 replication in ceph means pulling the OSD node would "gaurantee" > the pg's to go inactive - which could be ok - but there is nothing > gauranteeing that the other Hadoop replicas are not served out of the > same OSD-node/pg? In which case - rebooting an OSD node would bring the > hadoop cluster unavailable. > > Is anyone serving HBase out of Ceph - how does the stadck and > configuration look? If I went for 3 x replication in both Ceph and HDFS > then it would definately work, but 9x copies of the dataset is a bit > more than what looks feasible at the moment. > > Thanks for your reflections/input. > > Jesper > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

active+remapped+backfilling keeps going .. and going
by Kyriazis, George 27 Apr '20

27 Apr '20

Hello, I have a Proxmox ceph cluster with 5 nodes and 3 OSDs each (total 15 OSDs), on a 10G network. The cluster started small, and I’ve progressively added OSDs over time. Problem is…. The cluster never rebalances completely. There is always progress on backfilling, but PGs that used to be in active+clean state jump back into the active+remapped+backfilling (or active+remapped+backfill_wait) state, to be moved to different OSDs. Initially I had a 1G network (recently upgraded to 10G), and I was holding on the backfill settings (osd_max_backfills and osd_recovery_sleep_hdd). I just recently (last few weeks) upgraded to 10G, with osd_max_backfills = 50 and osd_recovery_sleep_hdd = 0 (only HDDs, no SSDs). Cluster has been backfilling for months now with no end in sight. Is this normal behavior? Is there any setting that I can look at that till give me an idea as to why PGs are jumping back into remapped from clean? Below is output of “ceph osd tree” and “ceph osd df”: # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72472 root default -9 40.01666 host vis-hsw-01 3 hdd 10.91309 osd.3 up 1.00000 1.00000 6 hdd 14.55179 osd.6 up 1.00000 1.00000 10 hdd 14.55179 osd.10 up 1.00000 1.00000 -13 40.01666 host vis-hsw-02 0 hdd 10.91309 osd.0 up 1.00000 1.00000 7 hdd 14.55179 osd.7 up 1.00000 1.00000 11 hdd 14.55179 osd.11 up 1.00000 1.00000 -11 40.01666 host vis-hsw-03 4 hdd 10.91309 osd.4 up 1.00000 1.00000 8 hdd 14.55179 osd.8 up 1.00000 1.00000 12 hdd 14.55179 osd.12 up 1.00000 1.00000 -3 40.01666 host vis-hsw-04 5 hdd 10.91309 osd.5 up 1.00000 1.00000 9 hdd 14.55179 osd.9 up 1.00000 1.00000 13 hdd 14.55179 osd.13 up 1.00000 1.00000 -15 43.65807 host vis-hsw-05 1 hdd 14.55269 osd.1 up 1.00000 1.00000 2 hdd 14.55269 osd.2 up 1.00000 1.00000 14 hdd 14.55269 osd.14 up 1.00000 1.00000 -5 0 host vis-ivb-07 -7 0 host vis-ivb-10 # # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 3 hdd 10.91309 1.00000 11 TiB 8.2 TiB 8.2 TiB 552 MiB 25 GiB 2.7 TiB 75.08 1.19 131 up 6 hdd 14.55179 1.00000 15 TiB 9.1 TiB 9.1 TiB 1.2 GiB 30 GiB 5.5 TiB 62.47 0.99 148 up 10 hdd 14.55179 1.00000 15 TiB 8.1 TiB 8.1 TiB 1.5 GiB 20 GiB 6.4 TiB 55.98 0.89 142 up 0 hdd 10.91309 1.00000 11 TiB 7.5 TiB 7.4 TiB 504 MiB 24 GiB 3.5 TiB 68.34 1.09 120 up 7 hdd 14.55179 1.00000 15 TiB 8.7 TiB 8.7 TiB 1.0 GiB 31 GiB 5.8 TiB 60.07 0.95 144 up 11 hdd 14.55179 1.00000 15 TiB 9.4 TiB 9.3 TiB 819 MiB 20 GiB 5.2 TiB 64.31 1.02 147 up 4 hdd 10.91309 1.00000 11 TiB 7.0 TiB 7.0 TiB 284 MiB 25 GiB 3.9 TiB 64.35 1.02 112 up 8 hdd 14.55179 1.00000 15 TiB 9.3 TiB 9.2 TiB 1.8 GiB 29 GiB 5.3 TiB 63.65 1.01 157 up 12 hdd 14.55179 1.00000 15 TiB 8.6 TiB 8.6 TiB 623 MiB 19 GiB 5.9 TiB 59.14 0.94 136 up 5 hdd 10.91309 1.00000 11 TiB 8.6 TiB 8.6 TiB 542 MiB 29 GiB 2.3 TiB 79.01 1.26 134 up 9 hdd 14.55179 1.00000 15 TiB 8.2 TiB 8.2 TiB 707 MiB 27 GiB 6.3 TiB 56.56 0.90 138 up 13 hdd 14.55179 1.00000 15 TiB 8.7 TiB 8.7 TiB 741 MiB 18 GiB 5.8 TiB 59.85 0.95 134 up 1 hdd 14.55269 1.00000 15 TiB 9.8 TiB 9.8 TiB 1.3 GiB 20 GiB 4.8 TiB 67.18 1.07 158 up 2 hdd 14.55269 1.00000 15 TiB 8.7 TiB 8.7 TiB 936 MiB 18 GiB 5.8 TiB 60.04 0.95 148 up 14 hdd 14.55269 1.00000 15 TiB 8.3 TiB 8.3 TiB 673 MiB 18 GiB 6.3 TiB 56.97 0.90 131 up TOTAL 204 TiB 128 TiB 128 TiB 13 GiB 350 GiB 75 TiB 62.95 MIN/MAX VAR: 0.89/1.26 STDDEV: 6.44 # Thank you! George

5 9

Re: rbd perf image iostat - rbdmap write latency
by Jason Dillaman 27 Apr '20

27 Apr '20

On Mon, Apr 27, 2020 at 7:38 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > I guess this is not good for ssd (samsung sm863)? Or do I need to devide > 14.8 by 40? > The 14.8 ms number is the average latency coming from the OSDs, so no need to divide the number by anything. What is the size of your writes? At 40 writes/sec against an SSD-backed cluster, I can only hope they are large IOs. > > rbd perf image iostat > > NAME WR RD WR_BYTES RD_BYTES WR_LAT > RD_LAT > rbd.ssd/vps-test 40/s 0/s 5.0 MiB/s 0 B/s 14.84 ms > 0.00 ns > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Jason

1 0

rbd perf image iostat - rbdmap write latency
by Marc Roos 27 Apr '20

27 Apr '20

I guess this is not good for ssd (samsung sm863)? Or do I need to devide 14.8 by 40? rbd perf image iostat NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT rbd.ssd/vps-test 40/s 0/s 5.0 MiB/s 0 B/s 14.84 ms 0.00 ns

1 0

Not able to start object gateway
by Sailaja Yedugundla 27 Apr '20

27 Apr '20

I am trying to setup a single node cluster using cephadm. I was able to start the cluster with 1 monitor and 3 osds. When I try to create users for rados gateway using the command radosgw-admin user create, the command hangs. Here is my cluster status. cluster: id: 5a03d7e2-85e4-11ea-bba9-021f94750a41 health: HEALTH_WARN Reduced data availability: 44 pgs inactive Degraded data redundancy: 6/15 objects degraded (40.000%), 2 pgs degraded, 80 pgs undersized services: mon: 1 daemons, quorum ip-172-31-9-253.ec2.internal (age 9h) mgr: ip-172-31-9-253.ec2.internal.umudgc(active, since 9h) osd: 3 osds: 3 up (since 74m), 3 in (since 74m); 41 remapped pgs data: pools: 4 pools, 97 pgs objects: 5 objects, 1.2 KiB usage: 3.0 GiB used, 27 GiB / 30 GiB avail pgs: 45.361% pgs not active 6/15 objects degraded (40.000%) 2/15 objects misplaced (13.333%) 44 undersized+peered 24 active+undersized+remapped 17 active+clean 10 active+undersized 2 active+undersized+degraded progress: Rebalancing after osd.1 marked in (74m) [=========...................] (remaining: 2h) Rebalancing after osd.2 marked in (74m) [............................] Can someone help me resolving this issue.

2 2

Device /dev/rbd0 excluded by a filter.
by Marc Roos 27 Apr '20

27 Apr '20

Why is this info not available??? https://access.redhat.com/solutions/4009431

2 5

best hair transplant clinic in bangalore
by maachairtransplant123＠gmail.com 27 Apr '20

27 Apr '20

Best hair transplant in bangalore:Get advanced hair loss treatment by best Hair specialists in Bangalore! We have award winning doctors for Hair transplantation. Contact: https://www.maachairtransplantbangalore.com/

1 0

Sporadic mgr segmentation fault
by XuYun 26 Apr '20

26 Apr '20

Dear ceph users, We are experiencing sporadic mgr crash in all three ceph clusters (version 14.2.6 and version 14.2.8), the crash log is: 2020-04-17 23:10:08.986 7fed7fe07700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: In function 'const char* ceph::buffer::v14_2_0::ptr::c_str() const' thread 7fed7fe07700 time 2020-04-17 23:10:08.984887 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: 578: FAILED ceph_assert(_raw) ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fed8605c325] 2: (()+0x2534ed) [0x7fed8605c4ed] 3: (()+0x5a21ed) [0x7fed863ab1ed] 4: (PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)+0xbd) [0x7fed863840ed] 5: (AsyncConnection::_try_send(bool)+0xb6) [0x7fed8632fc76] 6: (ProtocolV2::write_message(Message*, bool)+0x832) [0x7fed8635bf52] 7: (ProtocolV2::write_event()+0x175) [0x7fed863718c5] 8: (AsyncConnection::handle_write()+0x40) [0x7fed86332600] 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x1397) [0x7fed8637f997] 10: (()+0x57c977) [0x7fed86385977] 11: (()+0x80bdaf) [0x7fed86614daf] 12: (()+0x7e65) [0x7fed8394ce65] 13: (clone()+0x6d) [0x7fed825fa88d] 2020-04-17 23:10:08.990 7fed7ee05700 -1 *** Caught signal (Segmentation fault) ** in thread 7fed7ee05700 thread_name:msgr-worker-2 ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) 1: (()+0xf5f0) [0x7fed839545f0] 2: (ceph::buffer::v14_2_0::ptr::release()+0x8) [0x7fed863aafd8] 3: (ceph::crypto::onwire::AES128GCM_OnWireTxHandler::~AES128GCM_OnWireTxHandler()+0x59) [0x7fed86388669] 4: (ProtocolV2::reset_recv_state()+0x11f) [0x7fed8635f5af] 5: (ProtocolV2::stop()+0x77) [0x7fed8635f857] 6: (ProtocolV2::handle_existing_connection(boost::intrusive_ptr<AsyncConnection>)+0x5ef) [0x7fed86374f8f] 7: (ProtocolV2::handle_client_ident(ceph::buffer::v14_2_0::list&)+0xd9c) [0x7fed8637673c] 8: (ProtocolV2::handle_frame_payload()+0x1fb) [0x7fed86376c1b] 9: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x7fed86376e70] 10: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)+0x44d) [0x7fed863773cd] 11: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x34) [0x7fed86360534] 12: (AsyncConnection::process()+0x186) [0x7fed86330656] 13: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7fed8637f015] 14: (()+0x57c977) [0x7fed86385977] 15: (()+0x80bdaf) [0x7fed86614daf] 16: (()+0x7e65) [0x7fed8394ce65] 17: (clone()+0x6d) [0x7fed825fa88d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Any thoughts about this issue? Xu Yun

2 4

Is "." a legal characted in device class or not?
by Frank Schilder 25 Apr '20

25 Apr '20

Dear all, I created https://tracker.ceph.com/issues/45253 on an inconsistency in allowed characters in device class names. Short story: I created a device class with name "rbd.meta" and all worked fine until I did [root@gnosis ~]# ceph osd crush class ls-osd rbd.meta Invalid command: invalid chars . in rbd.meta osd crush class ls-osd <class> : list all osds belonging to the specific <class> Error EINVAL: invalid command Question is now, is "." legal or should I change? Note: I really like the dot notation here and everything seems to work as expected, except the "class ls-osd" command. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Newbie question - how do I add more monitors?
by c.stoddart＠sheffield.ac.uk 24 Apr '20

24 Apr '20

version 14.2.9 nautilus I have just set up my first cluster with one admin/monitor host and six OSDs. Everything appears to be working fine. I wanted to add a couple more monitors on the OSDs, and following the quick ceph deploy documentation I did: ceph-deploy mon add icephnode01 ceph-deploy mon add icephnode04 This installs and starts the monitor daemons on the two nodes, but they are stuck at probing. They have the right networking and firewall exemptions as far as I can tell, so I don't think it's that. If I get a copy of the monmap with 'ceph mon getmap' and use monmaptool to look at it I can only see that one host. Likewise ceph.conf only contains the one initial host. Should I have done something before the ceph-deploy mon add to modify the monmap? Or am I way off base? Thanks for your patience with a beginner question! Chris

3 2

Check if upmap is supported by client?
by Frank Schilder 24 Apr '20

24 Apr '20

Dear all, I would like to enable the balancer on a mimic 13.2.8 cluster in upmap mode. Unfortunately, I have a lot of ceph fs kernel clients that report their version as jewel, but might already support upmap. The ceph client kernel module received already a lot of back-ports and supports features of later ceph versions, for example, quotas. I guess they report back jewel, because not all luminous/mimic features are back-ported yet. Is there a way to check if a client supports upmap? Here some info: [root@gnosis ~]# ceph features { [...] "client": [ { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 1676 }, { "features": "0x2f018fb86aa42ada", "release": "luminous", "num": 1 }, { "features": "0x3ffddff8ffacfffb", "release": "luminous", "num": 167 } ], The fs clients are the top two entries, the third entry is rbd clients. Note that the feature key for the fs clients is almost identical. Here a snippet from mds session ls for one such jewel client: { "id": 25641514, "num_leases": 0, "num_caps": 1, "state": "open", "request_load_avg": 0, "uptime": 588563.550276, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, "inst": "client.25641514 192.168.57.124:0/3398308464", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn253.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/groups" } } Since I would like to use upmap right from the beginning, my alternative is to re-weight a few of the really bad outliers manually to simplify changing back. What would you suggest? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 4

Repeatedly OSD crashes in PrimaryLogPG::hit_set_trim()
by KOT MATPOCKuH 24 Apr '20

24 Apr '20

Hello all! I'm got repeatedlyOSD crashes for 3 of OSD which stracktrace: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) 1: (()+0x911e70) [0x564d0067fe70] 2: (()+0xf5d0) [0x7f1272dad5d0] 3: (gsignal()+0x37) [0x7f1271dce2c7] 4: (abort()+0x148) [0x7f1271dcf9b8] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x242) [0x7f12762252b2] 6: (()+0x25a337) [0x7f1276225337] 7: (PrimaryLogPG::hit_set_trim(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >&, unsigned int)+0x930) [0x564d002ab480] 8: (PrimaryLogPG::hit_set_persist()+0xa0c) [0x564d002afafc] 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2989) [0x564d002c5f09] 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xc99) [0x564d002cac09] 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b7) [0x564d00124c87] 12: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x564d0039d8c2] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x592) [0x564d00144ae2] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3d3) [0x7f127622aec3] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f127622bab0] 16: (()+0x7dd5) [0x7f1272da5dd5] 17: (clone()+0x6d) [0x7f1271e95f6d] I think this happens after some network troubles and one PG was marked recovery_unfound, to solve this problem this command was executed: ceph pg 2.f8 mark_unfound_lost revert Also last lines in debug output before crash are related to this PG: -10001> 2020-04-23 14:16:18.790 7f534915a700 10 osd.12 pg_epoch: 10476 pg[2.f8( v 8673'30651498 (6953'30648450,8673'30651498] local-lis/les=10475/10476 n=14 ec=66/66 lis/c 10475/10427 les/c/f 10476/10428/201 10475/10475/10473) [12,13,17] r=0 lpr=10475 pi=[10427,10475)/2 crt=8673'30651498 lcod 0'0 mlcod 0'0 active mbc={}] get_object_context: obc NOT found in cache: 2:1f000000:.ceph-internal::hit_set_2.f8_archive_2020-04-22 02%3a57%3a10.496532Z_2020-04-22 03%3a57%3a11.211949Z:head -10001> 2020-04-23 14:16:18.790 7f534915a700 10 osd.12 pg_epoch: 10476 pg[2.f8( v 8673'30651498 (6953'30648450,8673'30651498] local-lis/les=10475/10476 n=14 ec=66/66 lis/c 10475/10427 les/c/f 10476/10428/201 10475/10475/10473) [12,13,17] r=0 lpr=10475 pi=[10427,10475)/2 crt=8673'30651498 lcod 0'0 mlcod 0'0 active mbc={}] get_object_context: no obc for soid 2:1f000000:.ceph-internal::hit_set_2.f8_archive_2020-04-22 02%3a57%3a10.496532Z_2020-04-22 03%3a57%3a11.211949Z:head and !can_create I'm evicted must of PG's from cache pool, and currently we have only two rados objects in this PG: # rados --pgid 2.f8 ls rbd_data.10d4416b8b4567.0000000000001dfb rbd_header.1db946b8b4567 I tried to remove one of them: rados -p vms-cache rm rbd_header.1db946b8b4567 But command not completed for last ~8 hours, because repeatedly OSD crashes. Does we have any ways to solve this problem without data loss? -- MATPOCKuH

1 0

v13.2.10 Mimic released
by Abhishek Lekshmanan 23 Apr '20

23 Apr '20

We're happy to announce the availability of the tenth bugfix release of Ceph Mimic, this release fixes a RGW vulnerability affecting mimic, and we recommend that all mimic users upgrade. Notable Changes --------------- * CVE 2020 12059: Fix an issue with Post Object Requests with Tagging (#44967, Lei Cao, Abhishek Lekshmanan) Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-13.2.10.tar.gz * For packages: http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 564bdc4ae87418a232fc901524470e1a0f76d641 * Release blog: https://ceph.io/releases/v13-2-10-mimic-released -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer

1 0

Increase number of read and writes
by Bobby 23 Apr '20

23 Apr '20

Hi, I am using Ceph in developer mode. Currently I am implementing Librados examples which are also available in Introduction to Librados section https://docs.ceph.com/docs/master/rados/api/librados-intro/#step-3-creating…. It says once your app has a cluster handle and a connection to a Ceph Storage Cluster, you may create an I/O Context and begin reading and writing data. For example, *err = rados_write(io, "hw", "Hello World!", 12, 0); if (err < 0) { fprintf(stderr, "%s: Cannot write object \"neo-obj\" to pool %s: %s\n", argv[0], poolname, strerror(-err)); rados_ioctx_destroy(io); rados_shutdown(cluster); exit(1); } else { printf("\nWrote \"Hello World\" to object \"neo-obj\".\n"); }* My question, Is "12" is the number of writes? Because I want to test the with high number of read and writes. Looking for help !

2 2

Dear Abby: Why Is Architecting CEPH So Hard?
by Linus VanWeil 23 Apr '20

23 Apr '20

Hello, Looks like the original chain got deleted, but thank you to everyone who responded. Just to keep any new-comers in the loop, I have pasted the original positing below. To all the original contributors to this chain, I feel much more confident in my design theory for the storage nodes. However, I wanted to narrow the focus and see if I can get any elaborated comments on the two below topics. Does anyone have any real-world data on metrics I can use to size MONs? When are they active? When do they utilize CPU, RAM, Storage (ie. larger storage pools require more resources, resources are used during recovery, etc.)? For anyone that commented or has opinions on Storage node sizing: How does choosing EC vs 3X replication affect your sizing of CPU / RAM? IS the some kind of over-head generalization I can use if assuming EC (ie. add an extra core per OSD)? I understand that recoveries are where this is most important, so I am looking for sizing metrics based on living through worst case scenarios. --------------------------- ORIGINAL POSTING: Hey Folks, This is my first ever post here in the CEPH user group and I will preface with the fact that I know this is a lot of what many people ask frequently. Unlike what I assume to be a large majority of CEPH “users” in this forum, I am more of a CEPH “distributor.” My interests lie in how to build a CEPH environment to best fill an organization’s needs.I am here for the real-world experience and expertise so that I can learn to build CEPH “right.” I have spent the last couple years collecting data on general “best practices” through forum posts, CEPH documentation, CEPHLACON, etc. I wanted to post my findings to the forum to see where I can harden my stance. Below are two example designs that I might use when architecting a solution currently. I have specific questions around design elements in each that I would like you to approve for holding water or not. I want to focus on the hardware, so I am asking for generalizations where possible. Let’s assume in all scenarios that we are using Luminous and that the data type is mixed use. I am not expecting anyone to run through every question, so please feel free to comment on any piece you can. Tell me what is overkill and what is lacking! Example 1: 8x 60-Bay (8TB) Storage nodes (480x 8TB SAS Drives) Storage Node Spec: 2x 32C 2.9GHz AMD EPYC - Documentation mentions .5 cores per OSD for throughput optimized. Are they talking about .5 Physical cores or .5 Logical cores? - Is it better to pick my processors based on a total GHz measurement like 2GHz per OSD? - Would a theoretical 8C at 2GHz serve the same number of OSDs as a 16C at 1GHz? Would Threads be included in this calculation? 512GB Memory - I know this is the hot topic because of its role in recoveries. Basically, I am looking for the most generalized practice I can use as a safe number and a metric I can use as a nice to have. - Is it 1GB of RAM per TB of RAW OSD? 2x 3.2TB NVMe WAHLDB / Log Drive - Another hot topic that I am sure will bring many “it depends.” All I am looking for is experience on this. I know people have mentioned having at least 70GB of Flash for WAHLDB / Logs. - Can I use 70GB as a flat calculation per OSD or is it depend on the Size of the OSD? - I know more is better, but what is a number I can use to get started with minimal issues? 2x 56Gbit Links - I think this should be enough given the rule of thumb of 10Gbit for every 12 OSDs. 3x MON Node MON Node Spec: 1x 8C 3.2GHz AMD EPYC - I can’t really find good practices around when to increase your core count. Any suggestions? 128GB Memory - What do I need memory for in a MON? - When do I need to expand? 2x 480GB Boot SSDs - Any reason to look more closely into the sizing of these drives? 2x 25Gbit Uplinks - Should these match the output of the storage nodes for any reason? Example 2: 8x 12-Bay NVMe Storage nodes (96x 1.6TB NVMe Drives) Storage Node Spec: 2x 32C 2.9GHz AMD EPYC - I have read that each NMVe OSD should have 10 cores. I am not splitting Physical drives into multiple OSDs so let’s assume I have 12 OSD per Node. - Would threads count toward my 10 core quota or just physical cores? - Can I do a similar calculation as I mentioned before and just use 20GHz per OSD instead of focusing on cores specifically? 512GB Memory - I assume there is some reason I can’t use the same methodology of 1GB per TB of OSD since this is NVMe storage 2x 100Gbit Links - This is assuming about 1Gigabyte per second of real-world speed per disk 3x MON Node – What differences should MONs serving NVMe have compared to large NLSAS pools? MON Node Spec: 1x 8C 3.2GHz AMD Epyc 128GB Memory 2x 480GB Boot SSDs 2x 25Gbit Uplinks

1 0

Healthy objects trapped in incomplete pgs
by Jesper Lykkegaard Karlsen 23 Apr '20

23 Apr '20

Dear Cephers, A few days ago disaster struck the Ceph cluster (erasure-coded) I am administrating, as the UPS power was pull from the cluster causing a power outage. After rebooting the system, 6 osds were lost (spread over 5 osd nodes) as they could not mount anymore, several others had damages. This was more than the host-faliure domain was setup to handle and auto-recovery failed and osds started downing in a cascading maner. When the dust settled, there were 8 pgs (of 2048) inactive and a bunch of osds down. I managed to recover 5 pgs, mainly by ceph-objectstore-tool export/import/repair commands, but now I am left with 3 pgs that are inactive and incomplete. One of the pgs seems un-salvageable, as I cannot get to become active at all (repair/import/export/lowering min_size), but the two others I can get active if I export/import one of the pg shards and restart osd. Rebuilding then starts but after a while one of the osds holding the pgs goes down, with a "FAILED ceph_assert(clone_size.count(clone))" message in the log. If I set osds to noout nodown, then I can that it is only rather few objects e.g. 161 of a pg of >100000, that are failing to be remapped. Since most of the object in the two pgs seem intact, it would be sad to delete the whole pg (force-create-pg) and loose all that data. Is there a way to show and delete the failing objects? I have thought of a recovery plan and want to share that with you, so you can comment on this if it sounds doable or not? * Stop osds from recovering: ceph osd set norecover * bring back pgs active: ceph-objectstore-tool export/import and restart osd * find files in pgs: cephfs-data-scan pg_files <path> <pg id> * pull out as many as possible of those files to other location. * recreate pgs: ceph osd force-create-pg <pgid> * restart recovery: ceph osd unset norecover * copy back in the recovered files Would that work or do you have a better suggestion? Cheers, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka(a)mbg.au.dk Tlf: +45 50906203

1 0

missing amqp-exchange on bucket-notification with AMQP endpoint
by Andreas Unterkircher 23 Apr '20

23 Apr '20

Hello List, I'm trying to create a (S3-)bucket-notification into RabbitMQ via AMQP - on Ceph v15.2.1 octopus, using the official .deb packages on Debian Buster. I've created the following topic (directly via S3, not via pubsub REST API): <ListTopicsResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> <ListTopicsResult> <Topics> <member> <User>testuser</User> <Name>testtopic</Name> <EndPoint> <EndpointAddress>amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointAddress> <EndpointArgs>Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=amqp.direct&push-endpoint=amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointArgs> <EndpointTopic>testtopic</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:de::testtopic</TopicArn> <OpaqueData></OpaqueData> </member> </Topics> </ListTopicsResult> ... </ListTopicsResponse> Then I've created the following bucket-notification <NotificationConfiguration> <TopicConfiguration> <Id>notify-psapp</Id> <Topic>arn:aws:sns:de::testtopic</Topic> <Event>s3:ObjectCreated:*</Event> <Event>s3:ObjectRemoved:*</Event> </TopicConfiguration> </NotificationConfiguration> When I upload a file into the bucket, the event itself seems to get fired, but radosgw keeps tell me that cmqp-exchange is not set 2020-04-20T12:24:29.935+0200 7ff01c5d3700 1 ====== starting new request req=0x7ff01c5cad50 ===== 2020-04-20T12:24:30.019+0200 7ff01c5d3700 1 ERROR: failed to create push endpoint: amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672 due to: pubsub endpoint configuration error: AMQP: missing amqp-exchange But it's there in the EndpointArgs, right? Or do I miss it somewhere else? Best Regards, Andreas

2 4

How to remove a deamon from orch
by Ml Ml 22 Apr '20

22 Apr '20

Hello list, i somehow have this "mgr.cph02 ceph02 stopped " line here. root@ceph01:~# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID mgr.ceph02 ceph02 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 4e349a382c6b mgr.ceph03 ceph03 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 2a9a037e5e2d mgr.cph02 ceph02 stopped 2w ago - <unknown> <unknown> <unknown> <unknown> mon.ceph02 ceph02 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 cf66ca51c0dd mon.ceph03 ceph03 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 fceaaa03b41f I actually cant remember how i did that. How can i remove that wrong "mgr.cph02" entry? Thanks, Michael

1 0

Re: docs.ceph.com certificate expired?
by Jos Collin 22 Apr '20

22 Apr '20

Fixed On 22/04/20 6:57 pm, Bobby wrote: > > Thanks ! When will it be back? > > On Wed, Apr 22, 2020 at 3:03 PM <ulrich.weigand(a)de.ibm.com > <mailto:ulrich.weigand@de.ibm.com>> wrote: > > Hello, > > trying to access the documentation on docs.ceph.com > <http://docs.ceph.com> now results in an error: The certificate > expired on April 22, 2020, 8:46 AM. > > Bye, > Ulrich > _______________________________________________ > Dev mailing list -- dev(a)ceph.io <mailto:dev@ceph.io> > To unsubscribe send an email to dev-leave(a)ceph.io > <mailto:dev-leave@ceph.io> > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

1 0

block.db symlink missing after each reboot
by Stefan Priebe - Profihost AG 22 Apr '20

22 Apr '20

Hi there, i've a bunch of hosts where i migrated HDD only OSDs to hybird ones using: sudo -E -u ceph -- bash -c 'ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD} bluefs-bdev-new-db --dev-target /dev/bluefs_db1/db-osd${OSD}' while this worked fine and each OSD was running fine. It looses it's block.db symlink after reboot. If i manually recreate the block.db symlink inside: /var/lib/ceph/osd/ceph-* all osds start fine. Can anybody help who creates those symlinks and why they're not created automatically in case of migrated db? Greets, Stefan

3 5

MDS : replace a standby-replay daemon by an active one
by Herve Ballans 22 Apr '20

22 Apr '20

Hello to all confined people (and the others too) ! On one of my Ceph cluster (Nautilus 14.2.3), I previously set up 3 MDS daemons in active/standy-replay/standby configuration. For design reasons, I would like to replace this configuration by an active/active/standby one. It means replace the standby-replay daemon by an active one. I didn't find any clear procedure regarding this operation, and my question is about if I can add an active rank directly or if I have to unset the standby-replay status first ? I was thinking of the second option, that is: $ sudo ceph fs set /my_fs/ allow_standby_replay false $ sudo ceph fs set /my_fs/ max_mds 2 Is it the correct way ? Thanks in advance, Hervé

2 2

: nautilus : progress section in ceph status is stuck
by Vasishta Shastry 22 Apr '20

22 Apr '20

Hi, I upgraded a luminous cluster to nautilus and migrated Filestore OSD to bluestore using ceph-ansible playbook. I migrated 6 OSDs as of now, I see that since the last 3 days progress section of ceph status is stuck as below. Can anyone please help me to check what is going wrong ? progress: > Rebalancing after osd.6 marked out > [========================......] > Regards, Vasishta Shastry

3 2

certificate docs.ceph.com
by Nic De Muyer 22 Apr '20

22 Apr '20

Hi, It appears the certificate expired today for docs.ceph.com. Just thought I'd mention it here. kr, Nic De Muyer

1 0