May 2020 - ceph-users - lists.ceph.io

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay)
by Jake Grimmett 27 Aug '20

27 Aug '20

Dear all, After enabling "allow_standby_replay" on our cluster we are getting (lots) of identical errors on the client /var/log/messages like Apr 29 14:21:26 hal kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay) We are using the ml kernel 5.6.4-1.el7 on Scientific Linux 7.8 Cluster and client are running Ceph v14.2.9 Setting was enabled with: # ceph fs set cephfs allow_standby_replay true [root@ceph-s1 ~]# ceph mds stat cephfs:1 {0=ceph-s3=up:active} 1 up:standby-replay 2 up:standby Is this something to worry about, or should we just disable allow_standby_replay ? any advice appreciated, many thanks Jake Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

2 2

OSDs get full with bluestore logs
by Khodayar Doustar 21 Aug '20

21 Aug '20

Hi, I have a 3 node cluster of mimic with 9 osds (3 osds on each node). I use this cluster to test integration of an application with S3 api. The problem is that after a few days all OSD starts filling up with bluestore logs and goes down and out one by one! I cannot stop the logs and I cannot find the setting to fix this leakage, this should be a leakage in logs because it's not logical to fill up all OSD with bluefs logs. This is an example of logs which is being repeated in bluestore logs: [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path /var/lib/ceph/osd/ceph-5 . . . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 15:54:29.056488 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 15:54:30.057828 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 15:54:31.059252 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0xf5f0) [0x7f40fd47f5f0] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] 4: (BlueFS::log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x3b) [0x55603a1fad3b] 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x64) [0x55603a1da764] 6: (main()+0x2f3e) [0x55603a105e3e] 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] 8: (()+0x23115f) [0x55603a1d915f] 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- . . . And this is the output of daemonperf for one of remaining (filling up!) OSDs: [root@server2 ~]# ceph daemonperf osd.5 ------bluefs------- ------------bluestore------------- ----------osd----------- jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l rop | 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 ^C[root@server2 ~]# And this is the enospc logs when trying to start OSD: [root@server2 ~]# journalctl -u -f ceph-osd(a)4.service Failed to add match 'ceph-osd(a)4.service': Invalid argument Failed to add filters: Invalid argument [root@server2 ~]# journalctl -f -u ceph-osd(a)4.service -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10] Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon osd.4. Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon osd.4. and some more data: [root@server1 ~]# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 38 GiB 35 GiB 2.6 GiB 6.83 303 POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED .rgw.root 18 N/A N/A 1.1 KiB 0 50 GiB 4 4 63 B 4 B 1.1 KiB default.rgw.meta 19 N/A N/A 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182 B 9.0 KiB default.rgw.log 20 N/A N/A 0 B 0 50 GiB 107 107 307 KiB 205 KiB 0 B default.rgw.control 21 N/A N/A 0 B 0 50 GiB 5 5 0 B 0 B 0 B default.rgw.buckets.index 22 N/A N/A 0 B 0 50 GiB 5 5 501 B 60 B 0 B default.rgw.buckets.data 23 N/A N/A 377 MiB 0.39 50 GiB 160 160 385 B 1.1 KiB 603 MiB [root@server1 ~]# [root@server1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server1 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.0 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.1 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.2 -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 0 - host server2 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.3 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.4 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 1.04 99 osd.5 -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server3 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 0.96 78 osd.6 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.7 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.8 TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 MIN/MAX VAR: 0/1.04 STDDEV: 5.58 [root@server1 ~]# I'm kind of newbie to ceph, so any help or hint would be appreciated. Did I hit a bug or something is wrong with my configuration? Thanks a lot, Khodayar

5 5

Help
by Randy Morgan 17 Aug '20

17 Aug '20

We are seeking information on configuring Ceph to work with Noobaa and NextCloud. Randy -- Randy Morgan CSR Department of Chemistry/BioChemistry Brigham Young University randym(a)chem.byu.edu

3 2

Ceph Nautilus packages for Ubuntu Focal
by Stefan Kooman 17 Aug '20

17 Aug '20

Hi list, We're wondering if Ceph Nautilus packages will be provided for Ubuntu Focal Fossa (20.04)? You might wonder why one would not just use Ubuntu Bionic (18.04) instead of using the latest LTS. Here is why: a glibc bug in Ubuntu Bionic that *might* affect Open vSwitch (OVS) users [1]. We had quite a few issues with OVS deadlocks on hypervisors, and do not want to risk experiencing the same issues on our Ceph cluster(s). I'm not sure how many of you use OVS for bridging / bonding, but for those who do, running Ceph (Nautlilus / Octopus) on 20.04 would be preferred. Gr. Stefan [1]: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592

2 1

Ceph influxDB support versus Telegraf Ceph plugin?
by victorhooi＠yahoo.com 06 Aug '20

06 Aug '20

Hi, I've read that Ceph has some InfluxDB reporting capabilities inbuilt (https://docs.ceph.com/docs/master/mgr/influx/) However, Telegraf, which is the system reporting daemon for InfluxDB, also has a Ceph plugin (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph) Just curious what people's thoughts on the two are, or what they are using in production? Which is easier to deploy/maintain, have you found? Or more useful for alerting, or tracking performance gremlins? Thanks, Victor

2 2

IPv6 connectivity gone for Ceph Telemetry
by Wido den Hollander 24 Jul '20

24 Jul '20

Hi, I was just checking on a few (13) IPv6-only Ceph clusters and I noticed that they couldn't send their Telemetry data anymore: telemetry.ceph.com has address 8.43.84.137 This server used to have Dual-Stack connectivity while it was still hosted at OVH. It seemed to have moved to Red Hat, but lost IPv6 connectivity. How can we get this back? Wido

2 1

Problem with OSD::osd_op_tp thread had timed out and other connected issues
by Jan Pekař - Imatic 21 Jul '20

21 Jul '20

Hello, I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network Cluster was empty, fresh install. We filled cluster with data (small blocks) using RGW. Cluster is now used for testing so no client was using it during my admin operations mentioned below After a while (7TB of data / 40M objects uploaded) we decided, that we increase pg_num from 128 to 256 to better spread data and to speedup this operation, I've set ceph config set mgr target_max_misplaced_ratio 1 so that whole cluster rebalance as quickly as it can. I have 3 issues/questions below: 1) I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to restart with logged heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide timed out after 150 after a while OSD's were back so I continued after a while with my tests. My question - increasing number of PG with maximal target_max_misplaced_ratio was too much for that OSDs? It is not recommended to do it this way? I had no problem with this increase before, but configuration of cluster was slightly different and it was luminous version. 2) Rebuild was still slow so I increased number of backfills ceph tell osd.* injectargs "--osd-max-backfills 10" and reduced recovery sleep time ceph tell osd.* injectargs "--osd-recovery-sleep-hdd 0.01" and after few hours I noticed, that some of my OSD's were restarted during recovery, in log I can see ... |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 start_waiting_for_healthy | I observed network graph usage and network utilization was low during recovery (10Gbit was not saturated). So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that OSD is using threads and HDD timeouts are not influencing heartbeats to other OSD's and MON. It looks like it is not true. 3) After OSD was wrongly marked down I can see that cluster has object degraded. There were no degraded object before that. Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs degraded, 8 pgs undersized It means that this OSD disconnection causes data degraded? How is it possible, when no OSD was lost. Data should be on that OSD and after peering should be everything OK. With luminous I had no problem, after OSD up degraded objects where recovered/found during few seconds and cluster was healthy within seconds. Thank you very much for additional info. I can perform additional tests you recommend because cluster is used for testing purpose now. With regards Jan Pekar -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz | +420326555326 ============ --

4 7

bluestore_default_buffered_write = true
by Adam Koczarski 20 Jul '20

20 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

Poor Windows performance on ceph RBD.
by Frank Schilder 13 Jul '20

13 Jul '20

Dear all, maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a back-end store. We have a pool of spinning disks to create large low-demand data disks, mainly for backups and other cold storage. Everything is fine when using linux VMs. However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly created linux VM. If anyone has pointers what to look for, we would be very grateful. The OpenNebula installation is more or less default. The current OS and libvirt versions we use are: Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64 libvirt-client.x86_64 4.5.0-23.el7_7.1 @updates qemu-kvm-ev.x86_64 10:2.12.0-33.1.el7 @centos-qemu-ev Some benchmark results from good to worse workloads: rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 : 450MB/s rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 230MB/s rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 190MB/s rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 150MB/s rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 : 26MB/s dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB. We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to 1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s and does not recover. It is almost as if Windows translates large sequential writes to small random writes. If anyone has seen and solved this before, please let us know. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 9

OSDs taking too much memory, for buffer_anon
by Harald Staub 07 Jul '20

07 Jul '20

As a follow-up to our recent memory problems with OSDs (with high pglog values: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LJPJZPBSQRJ… ), we also see high buffer_anon values. E.g. more than 4 GB, with "osd memory target" set to 3 GB. Is there a way to restrict it? As it is called "anon", I guess that it would first be necessary to find out what exactly is behind this? Well maybe it is just as Wido said, with lots of small objects, there will be several problems. Cheers Harry

4 8

ceph qos
by 展荣臻（信泰） 26 Jun '20

26 Jun '20

Hi everyone: There are two types of qos in ceph(one based on tokenbucket algorithm,another based on mclock ). Which one I can use in nautilus production environment ？Thank you

2 1

ERROR: osd init failed: (1) Operation not permitted
by Ml Ml 22 Jun '20

22 Jun '20

Hello List, first of all: Yes - i made mistakes. Now i am trying to recover :-/ I had a healthy 3 node cluster which i wanted to convert to a single one. My goal was to reinstall a fresh 3 Node cluster and start with 2 nodes. I was able to healthy turn it from a 3 Node Cluster to a 2 Node cluster. Then the problems began. I started to change size=1 and min_size=1. Health was okay until here. Then over sudden both nodes got fenced...one node refused to boot, mons where missing, etc...to make long story short, here is where i am right now: root@node03:~ # ceph -s cluster b3be313f-d0ef-42d5-80c8-6b41380a47e3 health HEALTH_WARN 53 pgs stale 53 pgs stuck stale monmap e4: 2 mons at {0=10.15.15.3:6789/0,1=10.15.15.2:6789/0} election epoch 298, quorum 0,1 1,0 osdmap e6097: 14 osds: 9 up, 9 in pgmap v93644673: 512 pgs, 1 pools, 1193 GB data, 304 kobjects 1088 GB used, 32277 GB / 33366 GB avail 459 active+clean 53 stale+active+clean root@node03:~ # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 32.56990 root default -2 25.35992 host node03 0 3.57999 osd.0 up 1.00000 1.00000 5 3.62999 osd.5 up 1.00000 1.00000 6 3.62999 osd.6 up 1.00000 1.00000 7 3.62999 osd.7 up 1.00000 1.00000 8 3.62999 osd.8 up 1.00000 1.00000 19 3.62999 osd.19 up 1.00000 1.00000 20 3.62999 osd.20 up 1.00000 1.00000 -3 7.20998 host node02 3 3.62999 osd.3 up 1.00000 1.00000 4 3.57999 osd.4 up 1.00000 1.00000 1 0 osd.1 down 0 1.00000 9 0 osd.9 down 0 1.00000 10 0 osd.10 down 0 1.00000 17 0 osd.17 down 0 1.00000 18 0 osd.18 down 0 1.00000 my main mistakes seemd to be: -------------------------------- ceph osd out osd.1 ceph auth del osd.1 systemctl stop ceph-osd@1 ceph osd rm 1 umount /var/lib/ceph/osd/ceph-1 ceph osd crush remove osd.1 As far as i can tell, ceph waits and needs data from that OSD.1 (which i removed) root@node03:~ # ceph health detail HEALTH_WARN 53 pgs stale; 53 pgs stuck stale pg 0.1a6 is stuck stale for 5086.552795, current state stale+active+clean, last acting [1] pg 0.142 is stuck stale for 5086.552784, current state stale+active+clean, last acting [1] pg 0.1e is stuck stale for 5086.552820, current state stale+active+clean, last acting [1] pg 0.e0 is stuck stale for 5086.552855, current state stale+active+clean, last acting [1] pg 0.1d is stuck stale for 5086.552822, current state stale+active+clean, last acting [1] pg 0.13c is stuck stale for 5086.552791, current state stale+active+clean, last acting [1] [...] SNIP [...] pg 0.e9 is stuck stale for 5086.552955, current state stale+active+clean, last acting [1] pg 0.87 is stuck stale for 5086.552939, current state stale+active+clean, last acting [1] When i try to start ODS.1 manually, i get: -------------------------------------------- 2020-02-10 18:48:26.107444 7f9ce31dd880 0 ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 10210 2020-02-10 18:48:26.134417 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) backend xfs (magic 0x58465342) 2020-02-10 18:48:26.184202 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is supported and appears to work 2020-02-10 18:48:26.184209 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2020-02-10 18:48:26.184526 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2020-02-10 18:48:26.184585 7f9ce31dd880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_feature: extsize is disabled by conf 2020-02-10 18:48:26.309755 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2020-02-10 18:48:26.633926 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.642185 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.664273 7f9ce31dd880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2020-02-10 18:48:26.732154 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for clients 2020-02-10 18:48:26.732163 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400 was 8705, adjusting msgr requires for mons 2020-02-10 18:48:26.732167 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for osds 2020-02-10 18:48:26.732179 7f9ce31dd880 0 osd.1 6002 load_pgs 2020-02-10 18:48:31.939810 7f9ce31dd880 0 osd.1 6002 load_pgs opened 53 pgs 2020-02-10 18:48:31.940546 7f9ce31dd880 -1 osd.1 6002 log_to_monitors {default=true} 2020-02-10 18:48:31.942471 7f9ce31dd880 1 journal close /var/lib/ceph/osd/ceph-1/journal 2020-02-10 18:48:31.969205 7f9ce31dd880 -1 ESC[0;31m ** ERROR: osd init failed: (1) Operation not permittedESC[0m Its mounted: /dev/sdg1 3.7T 127G 3.6T 4% /var/lib/ceph/osd/ceph-1 Is there any way i can get the OSD.1 back in? Thanks a lot, mario

2 1

MAX AVAIL goes up when I reboot an OSD node
by Boris Behrens 14 Jun '20

14 Jun '20

Dear people on this mailing list, I've got the "problem" that our MAX AVAIL value increases by about 5-10 TB when I reboot a whole OSD node. After the reboot the value goes back to normal. I would love to know WHY. Under normal circumstances I would ignore this behavior, but because I am very new to the whole ceph software I would like to know why stuff like this happens. What I read is, that this value will be calculated by the most filled OSD. I've set noout and norebalance while the node is offline and I unset both values after the reboot. We are currently on nautilus. Cheers and thanks in advance Boris

6 9

Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
by Dan van der Ster 10 Jun '20

10 Jun '20

This means it has been applied: # ceph osd dump -f json | jq .require_osd_release "nautilus" -- dan On Mon, Feb 17, 2020 at 11:10 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > How do you check if you issued this command in the past? > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Excessive write load on mons after upgrade > from 12.2.13 -> 14.2.7 > > Hi Peter, > > could be a totally different problem but did you run the command "ceph > osd require-osd-release nautilus" after the upgrade? > We had poor performance after upgrading to nautilus and running this > command fixed it. The same was reported by others for previous updates. > Here is my original message regarding this issue: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPV… > > We did not observe the master election problem though. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: Reducing RAM usage on production MDS
by Patrick Donnelly 10 Jun '20

10 Jun '20

On Wed, May 27, 2020 at 10:09 PM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Hi all, > > The single active MDS on one of our Ceph clusters is close to running out of RAM. > > MDS total system RAM = 528GB > MDS current free system RAM = 4GB > mds_cache_memory_limit = 451GB > current mds cache usage = 426GB This mds_cache_memory_limit is way too high for the available RAM. We normally recommend that your RAM be 150% of your cache limit but we lack data for such large cache sizes. > Presumably we need to reduce our mds_cache_memory_limit and/or mds_max_caps_per_client, but would like some guidance on whether it’s possible to do that safely on a live production cluster when the MDS is already pretty close to running out of RAM. > > Cluster is Luminous - 12.2.12 > Running single active MDS with two standby. > 890 clients > Mix of kernel client (4.19.86) and ceph-fuse. > Clients are 12.2.12 (398) and 12.2.13 (3) v12.2.12 has the changes necessary to throttle MDS cache size reduction. You should be able to reduce mds_cache_memory_limit to any lower value without destabilizing the cluster. > The kernel clients have stayed under “mds_max_caps_per_client”: “1048576". But the ceph-fuse clients appear to hold very large numbers according to the ceph-fuse asok. > e.g. > “num_caps”: 1007144398, > “num_caps”: 1150184586, > “num_caps”: 1502231153, > “num_caps”: 1714655840, > “num_caps”: 2022826512, This data from the ceph-fuse asok is actually the number of caps ever received, not the current number. I've created a ticket for this: https://tracker.ceph.com/issues/45749 Look at the data from `ceph tell mds.foo session ls` instead. > Dropping caches on the clients appears to reduce their cap usage but does not free up RAM on the MDS. The MDS won't free up RAM until the cache memory limit is reached. > What is the safest method to free cache and reduce RAM usage on the MDS in this situation (without having to evict or remount clients)? reduce mds_cache_memory_limit > I’m concerned that reducing mds_cache_memory_limit even in very small increments may trigger a large recall of caps and overwhelm the MDS. That used to be the case in older versions of Luminous but not any longer. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

dealing with spillovers
by thoralf schulze 08 Jun '20

08 Jun '20

hi there, trying to get around my head rocksdb spillovers and how to deal with them … in particular, i have one osds which does not have any pools associated (as per ceph pg ls-by-osd $osd ), yet it does show up in ceph health detail as: osd.$osd spilled over 2.9 MiB metadata from 'db' device (49 MiB used of 37 GiB) to slow device compaction doesn't help. i am well aware of https://tracker.ceph.com/issues/38745 , yet find it really counter-intuitive that an empty osd with a more-or-less optimal sized db volume can't fit its rockdb on the former. is there any way to repair this, apart from re-creating the osd? fwiw, dumping the database with ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-$osd dump > bluestore_kv.dump yields a file of less than 100mb in size. and, while we're at it, a few more related questions: - am i right to assume that the leveldb and rocksdb arguments to ceph-kvstore-tool are only relevant for osds with filestore-backend? - does ceph-kvstore-tool bluestore-kv … also deal with rocksdb-items for osds with bluestore-backend? thank you very much & with kind regards, thoralf.

4 5

Cephadm Hangs During OSD Apply
by m＠silvenga.com 06 Jun '20

06 Jun '20

Hi, trying to migrate a second ceph cluster to Cephadm. All the host successfully migrated from "legacy" except one of the OSD hosts (cephadm kept duplicating osd ids e.g. two "osd.5", still not sure why). To make things easier, we re-provisioned the node (reinstalled from netinstall, applied the same SaltStack traits as the other nodes, wiped the disks) and tried to use cephadm to setup the OSD's. So, orch correctly starts the provisioning processes (a docker container running ceph-volume is created). But the provisioning never completes (docker exec): # ps axu root 1 0.1 0.2 99272 22488 ? Ss 15:26 0:01 /usr/libexec/platform-python -s /usr/sbin/ceph-volume lvm batch --no-auto /dev/sdb /dev/sdc --dmcrypt --yes --no-systemd root 807 0.9 0.5 154560 44120 ? S<L 15:26 0:06 /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-851cae40-3270-45ea-b788-be6e05465e92/osd-data-e3157b54-f6b9-4ec9-ab12-e289f52c00a4 Afr6Ct-ok4h-pBEy-GfFF-xxYl-EKwi-cHhjZc # cat /var/log/ceph/ceph-volume.log Running command: /usr/sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/ceph-851cae40-3270-45ea-b788-be6e05465e92/osd-data-e3157b54-f6b9-4ec9-ab12-e289f52c00a4 Running command: /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-851cae40-3270-45ea-b788-be6e05465e92/osd-data-e3157b54-f6b9-4ec9-ab12-e289f52c00a4 Afr6Ct-ok4h-pBEy-GfFF-xxYl-EKwi-cHhjZc # docker ps 2956dec0450d ceph/ceph:v15 "/usr/sbin/ceph-volu…" 14 minutes ago Up 14 minutes condescending_nightingale # cat osd_spec_default.yaml service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: all: true encrypted: true It looks like cephadm hangs on luksOpen. Is this expected (encryption is mentioned to be supported, outside of no documentation)?

2 4

crashing OSDs: ceph_assert(h->file->fnode.ino != 1)
by Harald Staub 05 Jun '20

05 Jun '20

This is again about our bad cluster, with too much objects, and the hdd OSDs have a DB device that is (much) too small (e.g. 20 GB, i.e. 3 GB usable). Now several OSDs do not come up any more. Typical error message: /build/ceph-14.2.8/src/os/bluestore/BlueFS.cc: 2261: FAILED ceph_assert(h->file->fnode.ino != 1) Also just tried to add a few GB to the DB device (lvextend, ceph-bluestore-tool bluefs-bdev-expand), but this also crashes, also with this message. Options that helped us before (thanks Wido :-) do not help here, e.g. CEPH_ARGS="--bluestore-rocksdb-options compaction_readahead_size=0" ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-$OSD compact Any ideas that I could try to save these OSDs? Cheers Harry

3 10

ceph orch upgrade stuck at the beginning.
by Gencer W. Genç 04 Jun '20

04 Jun '20

Hi, I've 15.2.1 installed on all machines. On primary machine I executed ceph upgrade command: $ ceph orch upgrade start --ceph-version 15.2.2 When I check ceph -s I see this: progress: Upgrade to docker.io/ceph/ceph:v15.2.2 (30m) [=...........................] (remaining: 8h) It says 8 hours. It is already ran for 3 hours. No upgrade processed. It get stuck at this point. Is there any way to know why this has stuck? Thanks, Gencer.

3 19

Cephadm Setup Query
by Shivanshi . 04 Jun '20

04 Jun '20

2 1

Octopus 15.2.2 unable to make drives available (reject reason locked)...
by Marco Pizzolo 04 Jun '20

04 Jun '20

Hello, Hitting an issue with a new 15.2.2 deployment using cephadm. I am having a problem creating encrypted, 2 osds per device OSDs (they are NVMe). After removing and bootstrapping the cluster again, i am unable to create OSDs as they're locked. sgdisk, wipefs, zap all fail to leave the drives as available. Any help would be appreciated. Any comments on performance experiences with ceph in containers (cephadm deployed) vs bare metal (ceph-deploy) would be greatly appreciated as well. Thanks, Marco ceph orch device ls HOST PATH TYPE SIZE DEVICE AVAIL REJECT REASONS prdhcistonode01 /dev/nvme0n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_2006266528D1 False *locked* prdhcistonode01 /dev/nvme1n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_2006266534D9 False *locked* prdhcistonode01 /dev/nvme2n1 ssd 953G INTEL SSDPEKKF010T8_BTHH850215GA1P0E False *locked* prdhcistonode01 /dev/nvme3n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_200626651473 False *locked* prdhcistonode01 /dev/nvme4n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_2006266508FB False * locked* prdhcistonode01 /dev/nvme5n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_20062664E6E8 False *locked* prdhcistonode01 /dev/nvme6n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_200626653CC0 False * locked* prdhcistonode01 /dev/nvme7n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_1939243B797E False * locked* prdhcistonode01 /dev/nvme8n1 ssd 11.6T Micron_9300_MTFDHAL12T8TDR_200626652441 False *locked* lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme2n1 259:0 0 953.9G 0 disk ├─nvme2n1p1 259:1 0 512M 0 part /boot/efi └─nvme2n1p2 259:2 0 953.4G 0 part / nvme3n1 259:3 0 11.7T 0 disk └─ceph--5bd47cae--97b3--4cad--b010--215fd982497b-osd--data--e6045acd--a56d--41d2--a016--b8647b9a717a 253:1 0 11.7T 0 lvm nvme4n1 259:4 0 11.7T 0 disk └─ceph--bf7dbfb4--afe3--4391--9847--08e461bf6247-osd--data--12faafac--b695--4c30--b6d7--7046d8275d9f 253:0 0 11.7T 0 lvm nvme0n1 259:5 0 11.7T 0 disk └─ceph--1a5d8e23--ff7d--44c3--b6d2--de143fed2b7d-osd--block--b6593547--e99a--4add--8edd--5d0fb53254cd 253:2 0 11.7T 0 lvm nvme5n1 259:6 0 11.7T 0 disk └─ceph--7d85ff24--79c8--4792--a2c8--bb4908f77ff0-osd--data--fc4e9dbd--920f--41b8--8467--74e9dcbd57ca 253:3 0 11.7T 0 lvm nvme6n1 259:7 0 11.7T 0 disk └─ceph--d8c8652a--1cd8--4e10--a333--4ea10f3b5004-osd--data--9a70a549--3cba--4f0d--a13a--8465781a10e9 253:5 0 11.7T 0 lvm nvme8n1 259:8 0 11.7T 0 disk └─ceph--e1914f1c--2385--4c0c--9951--d4b9200b7164-osd--data--8876559c--6393--4fbc--821b--7ac74cfb5a54 253:7 0 11.7T 0 lvm nvme7n1 259:9 0 11.7T 0 disk └─ceph--3765b53a--75eb--489e--97e1--d6b03bc25532-osd--data--777638e0--a325--401d--a01d--459676871003 253:4 0 11.7T 0 lvm nvme1n1 259:10 0 11.7T 0 disk └─ceph--2124f206--2b50--41a1--8a3c--d47c1a909a3b-osd--block--88e4f1eb--73f4--4c83--b978--fe7cabc0c3e6 253:6 0 11.7T 0 lvm

2 2

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

Help! ceph-mon is blocked after shutting down and ip address changed
by occj＠qq.com 04 Jun '20

04 Jun '20

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) os :CentOS Linux release 7.7.1908 (Core) single node ceph cluster with 1 mon,1mgr,1 mds,1rgw and 12osds , but only cephfs is used. ceph -s is blocked after shutting down the machine (192.168.0.104), then ip address changed to 192.168.1.6 I created the monmap with monmap tool and update the ceph.conf , hosts file and then start ceph-mon. and the ceph-mon log: ... 2019-12-11 08:57:45.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1285.14s 2019-12-11 08:57:50.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1290.14s 2019-12-11 08:57:55.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1295.14s 2019-12-11 08:58:00.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1300.14s 2019-12-11 08:58:05.172 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1305.14s 2019-12-11 08:58:10.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1310.14s 2019-12-11 08:58:15.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1315.14s 2019-12-11 08:58:20.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1320.14s 2019-12-11 08:58:25.174 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1325.14s ... I changed IP back to 192.168.0.104 yeasterday, but all the same. # cat /etc/ceph/ceph.conf [client.libvirt] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor log file = /var/log/ceph/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor [client.rgw.ceph-node1.rgw0] host = ceph-node1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-node1.rgw0.log rgw frontends = beast endpoint=192.168.1.6:8080 rgw thread pool size = 512 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.1.0/24 fsid = e384e8e6-94d5-4812-bfbb-d1b0468bdef5 mon host = [v2:192.168.1.6:3300,v1:192.168.1.6:6789] mon initial members = ceph-node1 osd crush chooseleaf type = 0 osd pool default crush rule = -1 public network = 192.168.1.0/24 [osd] osd memory target = 7870655146

2 1

Radosgw PubSub Traffic
by Dustin Guerrero 02 Jun '20

02 Jun '20

Hey all, We’ve been running some benchmarks against Ceph which we deployed using the Rook operator in Kubernetes. Everything seemed to scale linearly until a point where I see a single OSD receiving much higher CPU load than the other OSDs (nearly 100% saturation). After some investigation we noticed a ton of pubsub traffic in the strace coming from the RGW pods like so: [pid 22561] sendmsg(77, {msg_name(0)=NULL, msg_iov(3)=[{"\21\2)\0\0\0\10\0:\1\0\0\10\0\0\0\0\0\10\0\0\0\0\0\0\20\0\0-\321\211K"..., 73}, {"\200\0\0\0pubsub.user.ceph-user-wwITOk"..., 314}, {"\0\303\34[\360\314\233\2138\377\377\377\377\377\377\377\377", 17}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL|MSG_MORE <unfinished …> I’ve checked other OSDs and only a single OSD receives these messages. I suspect its creating a bottleneck. Does anyone have an idea on why these are being generated or how to stop them? The pubsub sync module doesn’t appear to be enabled, and our benchmark is doing simple gets/puts/deletes. We’re running Ceph 14.2.5 nautilus Thank you!

2 2

Cache pools at or near target size but no evict happen
by icy chan 01 Jun '20

01 Jun '20

Hi, I had configured a cache tier with max object counts 500k. But no evict happens when the object counts hit the configured maximum. Anyone experienced this issue? What should I do? $ ceph health detail HEALTH_WARN 1 cache pools at or near target size CACHE_POOL_NEAR_FULL 1 cache pools at or near target size cache pool 'cached-hdd-cache' with 887.11k objects at/near target max 500k objects $ ceph df | grep -e "POOL\|cached-hdd" POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cached-hdd 24 1.4 TiB 1.52M 1.4 TiB 0.60 78 TiB cached-hdd-cache 25 842 GiB 887.14k 842 GiB 15.97 1.4 TiB $ ceph osd pool get cached-hdd-cache all size: 3 min_size: 1 pg_num: 128 pgp_num: 128 crush_rule: nvme-repl-rule hashpspool: true nodelete: false nopgchange: false nosizechange: false write_fadvise_dontneed: false noscrub: false nodeep-scrub: false hit_set_type: bloom hit_set_period: 1200 hit_set_count: 4 hit_set_fpp: 0.05 use_gmt_hitset: 1 target_max_objects: 500000 target_max_bytes: 1099511627776 cache_target_dirty_ratio: 0 cache_target_dirty_high_ratio: 0.7 cache_target_full_ratio: 0.9 cache_min_flush_age: 0 cache_min_evict_age: 0 min_read_recency_for_promote: 1 min_write_recency_for_promote: 1 fast_read: 0 hit_set_grade_decay_rate: 20 hit_set_search_last_n: 1 pg_autoscale_mode: warn Regs, Icy

2 4

cephfs - modifying the ceph.file.layout of existing files
by Andrej Filipcic 31 May '20

31 May '20

Hi, I have two directories, cache_fast and cache_slow, and I would like to move the least used files from fast to slow, aka, user side tiering. cache_fast is pinned to fast_data ssd pool, while cache_slow to hdd cephfs_data pool. $ getfattr -n ceph.dir.layout /ceph/grid/cache_fast getfattr: Removing leading '/' from absolute path names # file: ceph/grid/cache_fast ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=fast_data" $ getfattr -n ceph.dir.layout /ceph/grid/cache_slow getfattr: Removing leading '/' from absolute path names # file: ceph/grid/cache_slow ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data" "mv" from cache_fast dir to cache_slow dir only renames the file in mds, but does not involve migration to a different pool and changing the file layout. The only option I see at this point is to "cp" the file to a new dir and removing it from the old one, but this would involve client side operations and can be very slow. Is there any better way, that would work ceph server side? Best regards, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

2 3

CEPH daemons crashed continously
by Mosharaf Hossain 31 May '20

31 May '20

I have build CEPH Cluster using "ceph version 14.2.8 nautilus". We observe continuous crash of "MON & MGR" service and healing automatically and the following error has been generated for MON service. Though the problem heals automatically but we need to know the reason of the problem. "utsname_hostname": "mon1", "assert_msg": "/build/ceph-14.2.8/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7fbee0e3c700 time 2020-05-29 05:51:40.385068\n/build/ceph-14.2.8/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero())\n", "crash_id": "2020-05-28_23:51:40.391577Z_9fcf058c-d03a-41a2-91bb-293f0a7763bd", "assert_func": "ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)", "ceph_version": "14.2.8" } -- Regards, *Mosharaf Hossain* Product Development IT Division Bangladesh Export Import IT Division (BEXIMCO)

1 0

Re: RGW orphans search
by Andrei Mikhailovsky 30 May '20

30 May '20

Hi Manuel, Thanks for the tip. Do you know if the latest code has this bug fixed? I was planning to upgrade to the latest major. Cheers ----- Original Message ----- > From: "EDH" <mriosfer(a)easydatahost.com> > To: "Andrei Mikhailovsky" <andrei(a)arhont.com>, "ceph-users" <ceph-users(a)ceph.io> > Sent: Saturday, 30 May, 2020 14:45:44 > Subject: RE: RGW orphans search > Hi Andrei, > > Orphans find code is not running. Will be deprecated in next reléase maybe > 14.2.10 > > Check: https://docs.ceph.com/docs/master/radosgw/orphans/ > > Stop progress is bugged. > > You got the same issue than us, multiparts are not being clean due a sharding > bugs. > > Or fast solution for recover 100TB , s3cmd sync to a other bucket and them > delete the old bucket. > > Not transparent at all but Works. > > Other recomendation: disable Dynamic shard and put a fixed shard number at your > config. > > Regards > Manuel > > > -----Mensaje original----- > De: Andrei Mikhailovsky <andrei(a)arhont.com> > Enviado el: sábado, 30 de mayo de 2020 13:12 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] RGW orphans search > > Hello, > > I am trying to clean up some wasted space (about 1/3 of used space in the rados > pool is currently unaccounted for including the replication level). I've > started the search command 20 days ago ( radosgw-admin orphans find > --pool=.rgw.buckets --job-id=ophans_clean1 --yes-i-really-mean-it ) and it's > still showing me the same thing: > > [ > { > "orphan_search_state": { > "info": { > "orphan_search_info": { > "job_name": "ophans_clean1", > "pool": ".rgw.buckets", > "num_shards": 64, > "start_time": "2020-05-10 21:39:28.913405Z" > } > }, > "stage": { > "orphan_search_stage": { > "search_stage": "iterate_bucket_index", > "shard": 0, > "marker": "" > } > } > } > } > ] > > > The output of the command keeps showing this (hundreds of thousands of lines): > > storing 1 entries at orphan.scan.ophans_clean1.linked.60 > > The total size of the pool is around 30TB and the buckets usage is just under > 10TB. The replica is 2. The activity on the cluster has spiked up since I've > started the command (currently seeing between 10-20K iops compared to a typical > 2-5k iops). > > Has anyone experienced this behaviour? It seems like the command should have > finished by now with only 30TB of used up space. I am running 13.2.10-1xenial > version of ceph. > > Cheers > > Andrei > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to > ceph-users-leave(a)ceph.io

1 0

Re: [ceph-users]: Ceph Nautius not working after setting MTU 9000
by Dave Hall 30 May '20

30 May '20

Hello. A few days ago I offered to share the notes I've compiled on network tuning. Right now it's a Google Doc: https://docs.google.com/document/d/1nB5fzIeSgQF0ti_WN-tXhXAlDh8_f8XF9GhU7J1… I've set it up to allow comments and I'd be glad for questions and feedback. If Google Docs not an acceptable format I'll try to put it up somewhere as HTML or Wiki. Disclosure: some sections were copied verbatim from other sources. Regarding the current discussion about iperf, the likely bottleneck is buffering. There is a per-NIC output queue set with 'ip link' and a per CPU core input queue set with 'sysctl'. Both should be set to some multiple of the frame size based on calculations related to link speed and latency. Jumping from 1500 to 9000 could negatively impact performance because one buffer or the other might be 1500 bytes short of a low multiple of 9000. It would be interesting to see the iperf tests repeated with corresponding buffer sizing. I will perform this experiment as soon as I complete some day-job tasks. -Dave Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 5/27/2020 6:51 AM, EDH - Manuel Rios wrote: > Anyone can share their table with other MTU values? > > Also interested into Switch CPU load > > KR, > Manuel > > -----Mensaje original----- > De: Marc Roos <M.Roos(a)f1-outsourcing.eu> > Enviado el: miércoles, 27 de mayo de 2020 12:01 > Para: chris.palmer <chris.palmer(a)pobox.com>; paul.emmerich <paul.emmerich(a)croit.io> > CC: amudhan83 <amudhan83(a)gmail.com>; anthony.datri <anthony.datri(a)gmail.com>; ceph-users <ceph-users(a)ceph.io>; doustar <doustar(a)rayanexon.ir>; kdhall <kdhall(a)binghamton.edu>; sstkadu <sstkadu(a)gmail.com> > Asunto: [ceph-users] Re: [External Email] Re: Ceph Nautius not working after setting MTU 9000 > > > Interesting table. I have this on a production cluster 10gbit at a > datacenter (obviously doing not that much). > > > [@]# iperf3 -c 10.0.0.13 -P 1 -M 9000 > Connecting to host 10.0.0.13, port 5201 > [ 4] local 10.0.0.14 port 52788 connected to 10.0.0.13 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 1.14 GBytes 9.77 Gbits/sec 0 690 KBytes > [ 4] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.08 MBytes > [ 4] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.08 MBytes > [ 4] 3.00-4.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.08 MBytes > [ 4] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.08 MBytes > [ 4] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.21 MBytes > [ 4] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.21 MBytes > [ 4] 7.00-8.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.21 MBytes > [ 4] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.21 MBytes > [ 4] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.21 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec 0 > sender > [ 4] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec > receiver > > > -----Original Message----- > Subject: Re: [ceph-users] Re: [External Email] Re: Ceph Nautius not > working after setting MTU 9000 > > To elaborate on some aspects that have been mentioned already and add > some others:: > > > * Test using iperf3. > > * Don't try to use jumbos on networks where you don't have complete > control over every host. This usually includes the main ceph network. > It's just too much grief. You can consider using it for limited-access > networks (e.g. ceph cluster network, hypervisor migration network, etc) > where you know every switch & host is tuned correctly. (This works even > when those nets share a vlan trunk with non-jumbo vlans - just set the > max value on the trunk itself, and individual values on each vlan.) > > * If you are pinging make sure it doesn't fragment otherwise you > will get misleading results: e.g. ping -M do -s 9000 x.x.x.x > * Do not assume that 9000 is the best value. It depends on your > NICs, your switch, kernel/device parameters, etc. Try different values > (using iperf3). As an example the results below are using a small cheap > Mikrotek 10G switch and HPE 10G NICs. It highlights how in this > configuration 9000 is worse than 1500, but that 5139 is optimal yet 5140 > is worst. The same pattern (obviously with different values) was > apparent when multiple tests were run concurrently. Always test your own > network in a controlled manner. And of course if you introduce anything > different later on, test again. With enterprise-grade kit this might not > be so common, but always test if you fiddle. > > > MTU Gbps (actual data transfer values using iperf3) - one particular > configuration only > > 9600 8.91 (max value) > 9000 8.91 > 8000 8.91 > 7000 8.91 > 6000 8.91 > 5500 8.17 > 5200 7.71 > 5150 7.64 > 5140 7.62 > 5139 9.81 (optimal) > 5138 9.81 > 5137 9.81 > 5135 9.81 > 5130 9.81 > 5120 9.81 > 5100 9.81 > 5000 9.81 > 4000 9.76 > 3000 9.68 > 2000 9.28 > 1500 9.37 (default) > > > Whether any of this will make a tangible difference for ceph is moot. I > just spend a little time getting the network stack correct as above, > then leave it. That way I know I am probably getting some benefit, and > not doing any harm. If you blindly change things you may well do harm > that can manifest itself in all sorts of ways outside of Ceph. Getting > some test results for this using Ceph will be easy; getting MEANINGFUL > results that way will be hard. > > > Chris > > > On 27/05/2020 09:25, Marc Roos wrote: > > > > > I would not call a ceph page, a random tuning tip. At least I hope > they > are not. NVMe-only with 100Gbit is not really a standard setup. I > assume > with such setup you have the luxury to not notice many > optimizations. > > What I mostly read is that changing to mtu 9000 will allow you to > better > saturate the 10Gbit adapter, and I expect this to show on a low end > busy > cluster. Don't you have any test results of such a setup? > > > > > -----Original Message----- > > Subject: Re: [ceph-users] Re: [External Email] Re: Ceph Nautius not > > working after setting MTU 9000 > > Don't optimize stuff without benchmarking *before and after*, don't > > apply random tuning tipps from the Internet without benchmarking > them. > > My experience with Jumbo frames: 3% performance. On a NVMe-only > setup > with 100 Gbit/s network. > > Paul > > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at > https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Tue, May 26, 2020 at 7:02 PM Marc Roos > <M.Roos(a)f1-outsourcing.eu> <mailto:M.Roos@f1-outsourcing.eu> > wrote: > > > > > Look what I have found!!! :) > https://ceph.com/geen-categorie/ceph-loves-jumbo-frames/ > > > > -----Original Message----- > From: Anthony D'Atri [mailto:anthony.datri@gmail.com] > Sent: maandag 25 mei 2020 22:12 > To: Marc Roos > Cc: kdhall; martin.verges; sstkadu; amudhan83; ceph-users; > doustar > Subject: Re: [ceph-users] Re: [External Email] Re: Ceph > Nautius not > > working after setting MTU 9000 > > Quick and easy depends on your network infrastructure. > Sometimes > it is > difficult or impossible to retrofit a live cluster without > disruption. > > > > On May 25, 2020, at 1:03 AM, Marc Roos > <M.Roos(a)f1-outsourcing.eu> <mailto:M.Roos@f1-outsourcing.eu> > > wrote: > > > > > > I am interested. I am always setting mtu to 9000. To be > honest I > > cannot imagine there is no optimization since you have less > interrupt > > requests, and you are able x times as much data. Every time > there > > > something written about optimizing the first thing mention > is > changing > > > to the mtu 9000. Because it is quick and easy win. > > > > > > > > > > -----Original Message----- > > From: Dave Hall [mailto:kdhall@binghamton.edu] > > Sent: maandag 25 mei 2020 5:11 > > To: Martin Verges; Suresh Rama > > Cc: Amudhan P; Khodayar Doustar; ceph-users > > Subject: [ceph-users] Re: [External Email] Re: Ceph Nautius > not > > working after setting MTU 9000 > > > > All, > > > > Regarding Martin's observations about Jumbo Frames.... > > > > I have recently been gathering some notes from various > internet > > sources regarding Linux network performance, and Linux > performance in > > general, to be applied to a Ceph cluster I manage but also > to the > rest > > > of the Linux server farm I'm responsible for. > > > > In short, enabling Jumbo Frames without also tuning a number > of > other > > kernel and NIC attributes will not provide the performance > increases > > we'd like to see. I have not yet had a chance to go through > the > rest > > of the testing I'd like to do, but I can confirm (via > iperf3) > that > > only enabling Jumbo Frames didn't make a significant > difference. > > > > Some of the other attributes I'm referring to are incoming > and > > outgoing buffer sizes at the NIC, IP, and TCP levels, > interrupt > > coalescing, NIC offload functions that should or shouldn't > be > turned > > on, packet queuing disciplines (tc), the best choice of TCP > slow-start > > > algorithms, and other TCP features and attributes. > > > > The most off-beat item I saw was something about adding > IPTABLES > rules > > > to bypass CONNTRACK table lookups. > > > > In order to do anything meaningful to assess the effect of > all of > > > these settings I'd like to figure out how to set them all > via > Ansible > > - so more to learn before I can give opinions. > > > > --> If anybody has added this type of configuration to Ceph > > Ansible, > > I'd be glad for some pointers. > > > > I have started to compile a document containing my notes. > It's > rough, > > > but I'd be glad to share if anybody is interested. > > > > -Dave > > > > Dave Hall > > Binghamton University > > > >> On 5/24/2020 12:29 PM, Martin Verges wrote: > >> > >> Just save yourself the trouble. You won't have any real > benefit > from > > MTU > >> 9000. It has some smallish, but it is not worth the effort, > > problems, > > and > >> loss of reliability for most environments. > >> Try it yourself and do some benchmarks, especially with > your > regular > >> workload on the cluster (not the maximum peak performance), > then > drop > > the > >> MTU to default ;). > >> > >> Please if anyone has other real world benchmarks showing > huge > > differences > >> in regular Ceph clusters, please feel free to post it here. > >> > >> -- > >> Martin Verges > >> Managing director > >> > >> Mobile: +49 174 9335695 > >> E-Mail: martin.verges(a)croit.io > >> Chat: https://t.me/MartinVerges > >> > >> croit GmbH, Freseniusstr. 31h, 81247 Munich > >> CEO: Martin Verges - VAT-ID: DE310638492 Com. register: > Amtsgericht > >> Munich HRB 231263 > >> > >> Web: https://croit.io > >> YouTube: https://goo.gl/PGE1Bx > >> > >> > >>> Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama > >> <sstkadu(a)gmail.com> <mailto:sstkadu@gmail.com> : > >> > >>> Ping with 9000 MTU won't get response as I said and it > should > be > > 8972. Glad > >>> it is working but you should know what happened to avoid > this > issue > > later. > >>> > >>>> On Sun, May 24, 2020, 3:04 AM Amudhan P > <amudhan83(a)gmail.com> <mailto:amudhan83@gmail.com> > wrote: > >>> > >>>> No, ping with MTU size 9000 didn't work. > >>>> > >>>> On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar > > <doustar(a)rayanexon.ir> <mailto:doustar@rayanexon.ir> > >>>> wrote: > >>>> > >>>>> Does your ping work or not? > >>>>> > >>>>> > >>>>> On Sun, May 24, 2020 at 6:53 AM Amudhan P > <amudhan83(a)gmail.com> <mailto:amudhan83@gmail.com> > > wrote: > >>>>> > >>>>>> Yes, I have set setting on the switch side also. > >>>>>> > >>>>>> On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, > > <doustar(a)rayanexon.ir> <mailto:doustar@rayanexon.ir> > >>>>>> wrote: > >>>>>> > >>>>>>> Problem should be with network. When you change MTU it > > should be > >>>> changed > >>>>>>> all over the network, any single hup on your network > should > > >>>>>>> speak > > and > >>>>>>> accept 9000 MTU packets. you can check it on your > hosts > with > >>> "ifconfig" > >>>>>>> command and there is also equivalent commands for > other > >>>> network/security > >>>>>>> devices. > >>>>>>> > >>>>>>> If you have just one node which it not correctly > configured > for > > MTU > >>>> 9000 > >>>>>>> it wouldn't work. > >>>>>>> > >>>>>>> On Sat, May 23, 2020 at 2:30 PM sinan(a)turka.nl > <sinan(a)turka.nl> <mailto:sinan@turka.nl> > >>> wrote: > >>>>>>>> Can the servers/nodes ping eachother using large > packet > sizes? > >>>>>>>> I > >>> guess > >>>>>>>> not. > >>>>>>>> > >>>>>>>> Sinan Polat > >>>>>>>> > >>>>>>>>> Op 23 mei 2020 om 14:21 heeft Amudhan P > <amudhan83(a)gmail.com> <mailto:amudhan83@gmail.com> > > het > >>>>>>>> volgende geschreven: > >>>>>>>>> In OSD logs "heartbeat_check: no reply from OSD" > >>>>>>>>> > >>>>>>>>>> On Sat, May 23, 2020 at 5:44 PM Amudhan P > > <amudhan83(a)gmail.com> <mailto:amudhan83@gmail.com> > >>>>>>>> wrote: > >>>>>>>>>> Hi, > >>>>>>>>>> > >>>>>>>>>> I have set Network switch with MTU size 9000 and > also in > my > >>> netplan > >>>>>>>>>> configuration. > >>>>>>>>>> > >>>>>>>>>> What else needs to be checked? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander > < > >>> wido(a)42on.com > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> On 5/23/20 12:02 PM, Amudhan P wrote: > >>>>>>>>>>>> Hi, > >>>>>>>>>>>> > >>>>>>>>>>>> I am using ceph Nautilus in Ubuntu 18.04 working > fine > wit > > MTU > >>>> size > >>>>>>>> 1500 > >>>>>>>>>>>> (default) recently i tried to update MTU size to > 9000. > >>>>>>>>>>>> After setting Jumbo frame running ceph -s is > timing > out. > >>>>>>>>>>> Ceph can run just fine with an MTU of 9000. But > there > is > >>> probably > >>>>>>>>>>> something else wrong on the network which is > causing > this. > >>>>>>>>>>> > >>>>>>>>>>> Check the Jumbo Frames settings on all the > switches as > well > > to > >>>> make > >>>>>>>> sure > >>>>>>>>>>> they forward all the packets. > >>>>>>>>>>> > >>>>>>>>>>> This is definitely not a Ceph issue. > >>>>>>>>>>> > >>>>>>>>>>> Wido > >>>>>>>>>>> > >>>>>>>>>>>> regards > >>>>>>>>>>>> Amudhan P > >>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To > >>>>>>>>>>>> unsubscribe send an email to > ceph-users-leave(a)ceph.io > >>>>>>>>>>>> > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To > unsubscribe > > >>>>>>>>>>> send an email to ceph-users-leave(a)ceph.io > >>>>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To > unsubscribe > >>>>>>>>> send an email to ceph-users-leave(a)ceph.io > >>>>>>>> _______________________________________________ > >>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To > unsubscribe > >>>>>>>> send an email to ceph-users-leave(a)ceph.io > >>>>>>>> > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io To > unsubscribe > send > >>>> an email to ceph-users-leave(a)ceph.io > >>>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io To > unsubscribe > send an > > >>> email to ceph-users-leave(a)ceph.io > >>> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io To > unsubscribe > send an > >> email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe > send > an > > email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe > send > an > > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 4

RGW orphans search
by Andrei Mikhailovsky 30 May '20

30 May '20

Hello, I am trying to clean up some wasted space (about 1/3 of used space in the rados pool is currently unaccounted for including the replication level). I've started the search command 20 days ago ( radosgw-admin orphans find --pool=.rgw.buckets --job-id=ophans_clean1 --yes-i-really-mean-it ) and it's still showing me the same thing: [ { "orphan_search_state": { "info": { "orphan_search_info": { "job_name": "ophans_clean1", "pool": ".rgw.buckets", "num_shards": 64, "start_time": "2020-05-10 21:39:28.913405Z" } }, "stage": { "orphan_search_stage": { "search_stage": "iterate_bucket_index", "shard": 0, "marker": "" } } } } ] The output of the command keeps showing this (hundreds of thousands of lines): storing 1 entries at orphan.scan.ophans_clean1.linked.60 The total size of the pool is around 30TB and the buckets usage is just under 10TB. The replica is 2. The activity on the cluster has spiked up since I've started the command (currently seeing between 10-20K iops compared to a typical 2-5k iops). Has anyone experienced this behaviour? It seems like the command should have finished by now with only 30TB of used up space. I am running 13.2.10-1xenial version of ceph. Cheers Andrei

2 1

warn if acting set violates failure domain
by Dan van der Ster 30 May '20

30 May '20

Hi, (In nautilus) I noticed that the acting set for a pg in acting+remapped+backfilling (or backfill_wait) can violate the failure domain rule. We have a 3x pool replicated across racks. Following some host outage I noticed several PGs like: 75.200 7570 0 7570 0 31583745536 0 0 3058 active+remapped+backfill_wait 55m 3208334'13480873 3208334:23611310 [596,502,717]p596 [596,502,424]p596 2020-05-28 05:44:42.604307 2020-05-28 05:44:42.604307 Checking those up and acting sets, I have: OK: PG 75.200 has no failure domain problem in up set [596, 502, 717] with racks ['BA09', 'BA10', 'BA12'] WARN: PG 75.200 has a failure domain problem in acting set [596, 502, 424] with racks ['BA09', 'BA10', 'BA09'] If BA09 were to go down, the pg would drop below min_size. (The full pg query output is at https://pastebin.com/rrEQMnyC) Is this https://tracker.ceph.com/issues/3360 ? # ceph osd dump | grep 75.200 pg_temp 75.200 [596,502,424] Shouldn't we mark a PG degraded or similar to raise HEALTH_WARN if the failure domain is violated? Cheers, Dan

1 0

v15.2.3 Octopus released
by Josh Durgin 30 May '20

30 May '20

We’re happy to announce the availability of the third Octopus stable release series. This release mainly is a workaround for a potential OSD corruption in v15.2.2. We advise users to upgrade to v15.2.3 directly. For users running v15.2.2 please execute the following:: ceph config set osd bluefs_preextend_wal_files false Changelog ~~~~~~~~~ * bluestore: common/options.cc: disable bluefs_preextend_wal_files Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: d289bbdec69ed7c1f516e0a093594580a76b78d0

1 0

OSD backups and recovery
by Ludek Navratil 30 May '20

30 May '20

HI all, what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local filesystem and backup the content using Amanda. I believe that there must a better way to do this but I couldn't find it in docs. I know that one option is to setup an archive zone, but it requires an additional ceph cluster that needs to be maintained and looked after. I would rather avoid that. How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me to recover the data correctly? Many thanks,Ludek

4 5

rocksdb tuning
by Frank R 29 May '20

29 May '20

Hi all, I am attempting to prevent bluestore rocksdb Level 3/4 spillover with a 150GB logical volume for the db/wal. I am thinking of setting max_bytes_for_level_base to about 1.3G (1342177280). This should let Level 3 fill up the 150GB logical volume. I don't expect to ever actually need L4. Another option would be to set max_bytes_for_level_base to 128m. This db/wal device is a fast SSD. OSD is HDD. Any opinions on whether it would be better to increase max_bytes_for_level_base to over 1G or decrease it to 128m? thanks, Frank

1 0

General question CephFS or RBD
by Willi Schiegel 29 May '20

29 May '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

2 1

Ceph and iSCSI
by Bobby 29 May '20

29 May '20

Hi all, I am new to Ceph. But I have a some good understanding of iSCSI protocol. I will dive into Ceph because it looks promising. I am particularly interested in Ceph-RBD. I have a request. Can you please tell me, if any, what are the common similarities between iSCSI and Ceph. If someone has to work on a common model for iSCSI and Ceph, what would be those significant points you would suggest to someone who has some understanding of iSCSI? Looking forward to answers. Thanks in advance :-) BR

2 1

Is 2 osds per disk, encryption possible with cephadm on 15.2.2?
by Marco Pizzolo 29 May '20

29 May '20

Hello Everyone, I'm still having issues getting the OSDs to properly create on a brand new Ceph 15.2.2 cluster. I don't see to be able to have OSDs created based on service definition of 2 osds per disk and encryption. It seems to hang and/or I see "No Deployments..." Has anyone had luck with this at this point? If no, how are you doing it? Thanks, Marco

1 0

CEPH failure domain - power considerations
by Phil Regnauld 29 May '20

29 May '20

Hi, in our production cluster, we have the following setup - 10 nodes - 3 drives / server (so far), mix of SSD and HDD (different pools) + NVMe - dual 10G in LACP, linked to two different switches (Cisco vPC) - OSDs, MONs and MGRs are colocated - A + B power feeds, 2 ATS (each receiving A+B) - ATS1 and ATS2 - 2 PDU rails, each connected to an ATS (PDU1 = ATS1, PDU2 = ATS2) - switches have dual PSUs and are connected to both rails - CEPH nodes - single power supply - Odd nodes (1,3,5...) are connected to PDU1 - Even nodes (2,4,6...) are connected to PDU2 ... I can provide a drawing if it helps :) Now, the default crush map ensures that multiple copies of the same object won't find their way on the same host, which is fine. But I'm thinking that in case of power failure [1] of either ATS or PDU, we'd be losing half the nodes in the cluster at the same time. How would I go about tuning our map so it took into account that, for a 3 copy replicated pool, we don't have those stored on hosts, say, 5,7,9 ? And, what about when using EC pools ? We currently have 5+2 SSD pools - how would we avoid losing availability in case of a power loss where 50% of the server are offline ? I've gone over https://docs.ceph.com/docs/master/rados/operations/crush-map/ but don't believe I'm at the stage where I dare make changes without incurring a huge data migration (probably can't be avoided). Any input appreciated. Cheers, Phil [1] both power feeds lost at the same time is really hard to protect against :)

8 10

Very bad performance on a ceph rbd pool via iSCSI to VMware esx
by Salsa 29 May '20

29 May '20

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 2

Ceph and Windows - experiences or suggestions
by Lars Täuber 29 May '20

29 May '20

Hi there! I got the task to connect a Windows client to our existing ceph cluster. I'm looking for experiences or suggestions from the community. There came two possibilities to my mind: 1. iSCSI Target on RBD exported to Windows 2. NFS-Ganesha on CephFS exported to Windows Is there a third way exporting a ceph cluster to a windows machine? I have some experiences with CephFS. We have a small cluster successfully running for linux clients. I don't have experiences with RBD or iSCSI. The Windows machine will use the space for backups. The kind of data is unknown. I expect the data to be a MS-SQL dump and user data from a sharepoint system. The Windows admin does not care whether NFS or iSCSI is used. I'd be happy if some of you could share experiences. Thanks Lars

10 10

Re: RBD Mirroring down+unknown
by Jason Dillaman 29 May '20

29 May '20

On Fri, May 29, 2020 at 12:09 PM Miguel Castillo <Miguel.Castillo(a)centro.net> wrote: > Happy New Year Ceph Community! > > I'm in the process of figuring out RBD mirroring with Ceph and having a > really tough time with it. I'm trying to set up just one way mirroring > right now on some test systems (baremetal servers, all Debian 9). The first > cluster is 3 nodes, and the 2nd cluster is 2 nodes (not worried about a > properly performing setup, just the functionality of RBD mirroring right > now). The purpose is to have a passive failover ceph cluster in a separate > DC. Mirroring seems like the best solution, but if we can't get it working, > we'll end up resorting to a scheduled rsync which is less than ideal. I've > followed several guides, read through a lot of documentation, and nothing > has worked for me thus far. If anyone can offer some troubleshooting help > or insight into what I might have missed in this setup, I'd greatly > appreciate it! I also don't fully understand the relationship between > images and pools and how you're supposed to configure statically sized > images for a pool that has a variable amount of dat > a, but that's a question for afterwards, I think :) > > Once RBD mirroring is set up, the mirror test image status shows as > down+unknown: > > On ceph1-dc2: > rbd --cluster dc1ceph mirror pool status fs_data --verbose > health: WARNING > images: 1 total > 1 unknown > > mirror_test: > global_id: c335017c-9b8f-49ee-9bc1-888789537c47 > state: down+unknown > description: status not found > last_update: > > Here are the commands I run using ceph-deploy on both clusters to get > everything up and running (run from a deploy directory on the first node of > each cluster). The clusters are created at the same time, and rbd setup > commands are only run after the clusters are up and healthy, and the > fs_data pool is created. > > ----------------------------------------------------------- > > Cluster 1 (dc1ceph): > > ceph-deploy new ceph1-dc1 ceph2-dc1 ceph3-dc1 > sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf > ceph-deploy install ceph1-dc1 ceph2-dc1 ceph3-dc1 --release luminous > ceph-deploy mon create-initial > ceph-deploy admin ceph1-dc1 ceph2-dc1 ceph3-dc1 > ceph-deploy mgr create ceph1-dc1 ceph2-dc1 ceph3-dc1 > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph1-dc1 ; done > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph2-dc1 ; done > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph3-dc1 ; done > ceph-deploy mds create ceph1-dc1 ceph2-dc1 ceph3-dc1 > ceph-deploy rgw create ceph1-dc1 ceph2-dc1 ceph3-dc1 > for f in 1 2 ; do scp ceph.client.admin.keyring > ceph$f-dc2:/etc/ceph/dc1ceph.client.admin.keyring ; done > for f in 1 2 ; do scp ceph.conf ceph$f-dc2:/etc/ceph/dc1ceph.conf ; done > for f in 1 2 ; do ssh ceph$f-dc2 "chown ceph.ceph /etc/ceph/dc1ceph*" ; > done > ceph osd pool create fs_data 512 512 replicated > rbd --cluster ceph mirror pool enable fs_data image > rbd --cluster dc2ceph mirror pool enable fs_data image > rbd --cluster ceph mirror pool peer add fs_data client.admin@dc2ceph > (generated id: b5e347b3-0515-4142-bc49-921a07636865) > rbd create fs_data/mirror_test --size=1G > rbd feature enable fs_data/mirror_test journaling > rbd mirror image enable fs_data/mirror_test > chown ceph.ceph ceph.client.admin.keyring > > Cluster 2 (dc2ceph): > > ceph-deploy new ceph1-dc2 ceph2-dc2 > sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf > ceph-deploy install ceph1-dc2 ceph2-dc2 --release luminous > ceph-deploy mon create-initial > ceph-deploy admin ceph1-dc2 ceph2-dc2 > ceph-deploy mgr create ceph1-dc2 ceph2-dc2 > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph1-dc2 ; done > for x in b c d e f g h i j k; do ceph-deploy osd create --data > /dev/sd${x}1 ceph2-dc2 ; done > ceph-deploy mds create ceph1-dc2 ceph2-dc2 > ceph-deploy rgw create ceph1-dc2 ceph2-dc2 > apt install rbd-mirror > for f in 1 2 3 ; do scp ceph.conf ceph$f-dc1:/etc/ceph/dc2ceph.conf ; done > for f in 1 2 3 ; do scp ceph.client.admin.keyring > ceph$f-dc1:/etc/ceph/dc2ceph.client.admin.keyring ; done > for f in 1 2 3 ; do ssh ceph$f-dc1 "chown ceph.ceph /etc/ceph/dc2ceph*" ; > done > ceph osd pool create fs_data 512 512 replicated > rbd --cluster ceph mirror pool peer add fs_data client.admin@dc1ceph > (generated id: e486c401-e24d-49bc-9800-759760822282) > systemctl enable ceph-rbd-mirror@admin > systemctl start ceph-rbd-mirror@admin > rbd --cluster dc1ceph mirror pool status fs_data --verbose > > > Cluster 1: > > ls /etc/ceph: > ceph.client.admin.keyring > ceph.conf > dc2ceph.client.admin.keyring > dc2ceph.conf > rbdmap > tmpG36OYs > > cat /etc/ceph/ceph.conf: > [global] > fsid = 8fede407-50e1-4487-8356-3dc98b30c500 > mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 > mon_host = *.*.*.1,*.*.*.27,*.*.*.41 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > cat /etc/ceph/dc2ceph.conf > [global] > fsid = 813ff410-02dc-47bd-b678-38add38495bb > mon_initial_members = ceph1-dc2, ceph2-dc2 > mon_host = *.*.*.56,*.*.*.0 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > > Cluster 2: > > ls /etc/ceph: > ceph.client.admin.keyring > ceph.conf > dc1ceph.client.admin.keyring > dc1ceph.conf > rbdmap > tmp_yxkPs > > cat /etc/ceph/ceph.conf > [global] > fsid = 813ff410-02dc-47bd-b678-38add38495bb > mon_initial_members = ceph1-dc2, ceph2-dc2 > mon_host = *.*.*.56,*.*.*.70 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > cat /etc/ceph/dc1ceph.conf > [global] > fsid = 8fede407-50e1-4487-8356-3dc98b30c500 > mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 > mon_host = *.*.*.1,*.*.*.27,*.*.*.41 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = *.*.*.0/24 > > > RBD Mirror daemon status: > > ceph-rbd-mirror(a)admin.service - Ceph rbd mirror daemon > Loaded: loaded (/lib/systemd/system/ceph-rbd-mirror@.service; enabled; > vendor preset: enabled) > Active: inactive (dead) since Mon 2020-01-06 16:21:44 EST; 3s ago > Process: 910178 ExecStart=/usr/bin/rbd-mirror -f --cluster ${CLUSTER} > --id admin --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS) > Main PID: 910178 (code=exited, status=0/SUCCESS) > > Jan 06 16:21:44 ceph1-dc2 systemd[1]: Started Ceph rbd mirror daemon. > Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462916 > 7f76ecf88780 -1 auth: unable to find a keyring on > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory > Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462949 > 7f76ecf88780 -1 monclient: ERROR: missing keyring, cannot use cephx for > authentication > Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: failed to initialize: (2) No > such file or directory2020-01-06 16:21:44.463874 7f76ecf88780 -1 > rbd::mirror::Mirror: 0x558d3ce6ce20 init: error connecting to local cluster > It seems like it's saying that "rbd-mirror" cannot access your keyring for DC2. Does the "ceph" user have read permission to the keyring? Can you run "sudo -u ceph ceph health" successfully? > > ------------------------------------------- > > I also tried running the ExecStart command manually, substituting in > different values for the parameters, and just never got it to work. If more > info is needed, please don't hesitate to ask. Thanks in advance! > > -Miguel > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Jason

1 0

bluestore - rocksdb level sizes
by Frank R 29 May '20

29 May '20

If I remember correctly, being able to configure the rocksdb level sizes was targeted for Octopus. I was wondering if this feature ever made it into the code as it would be useful when you want to use a drive smaller than 300G for the WAL/DB.

2 1

Virtual Ceph Days
by Mike Perez 29 May '20

29 May '20

Hello, Unfortunately, due to the COVID-19 pandemic, the Ceph Foundation is looking into running future Ceph Days virtually. We have created a survey to gather feedback from the community on how these events should run. You can access the survey here: https://survey.zohopublic.com/zs/jsCsIn We are interested in finding organizers to help with these events. Please opt-in the form if you're interested. -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 0

ceph with rdma can not mount with kernel
by 李亚锋 29 May '20

29 May '20

2 1

Ceph on CentOS 8?
by Jan Kasprzak 29 May '20

29 May '20

Hello, Ceph users, does anybody use Ceph on recently released CentOS 8? Apparently there are no el8 packages neither at download.ceph.com, nor in the native CentOS package tree. I am thinking about upgrading my cluster to C8 (because of other software running on it apart from Ceph). Do el7 packages simply work? Can they be rebuilt using rpmbuild --rebuild? Or is running Ceph on C8 more complicated than that? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | sir_clive> I hope you don't mind if I steal some of your ideas? laryross> As far as stealing... we call it sharing here. --from rcgroups

9 8

rbd image naming convention
by Palanisamy 29 May '20

29 May '20

Hello Team, We've integrated Ceph cluster storage with Kubernetes and provisioning volumes through rbd-provisioner. When we're creating volumes from yaml files in Kubernetes, pv > pvc > mounting to pod, In kubernetes end pvc are showing as meaningful naming convention as per yaml file defined. But in ceph cluster, rbd image name is creating with dynamic uid. During troubleshooting time, this will be tedious to find exact rbd image. Please find the provisioning logs in below pasted snippet. kubectl get pods,pv,pvc NAME READY STATUS RESTARTS AGE pod/sleepypod 1/1 Running 0 4m9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO Delete Bound default/test-dyn-pvc ceph-rbd 4m9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-dyn-pvc Bound pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO ceph-rbd 4m11s *rbd-provisioner logs* I1121 10:59:15.009012 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2" I1121 10:59:15.009092 1 controller.go:1087] provision "default/test-dyn-pvc" class "ceph-rbd": volume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" provisioned I1121 10:59:15.009138 1 controller.go:1101] provision "default/test-dyn-pvc" class "ceph-rbd": trying to save persistentvvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" I1121 10:59:15.020418 1 controller.go:1108] provision "default/test-dyn-pvc" class "ceph-rbd": persistentvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" saved I1121 10:59:15.020476 1 controller.go:1149] provision "default/test-dyn-pvc" class "ceph-rbd": succeeded I1121 10:59:15.020802 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-dyn-pvc", UID:"cd37d2d6-cecc-4a05-9736-c8d80abde7f5", APIVersion:"v1", ResourceVersion:"24545639", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 *rbd image details in Ceph cluster end* rbd -p kube ls --long NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2 1 GiB 2 is there way to setup proper naming convention for rbd image as well during kubernetes deployment itself. Kubernetes version: v1.15.5 Ceph cluster version: 14.2.2 nautilus (stable) *Best Regards,* *Palanisamy*

2 2

No scrubbing during upmap balancing
by Vytenis A 29 May '20

29 May '20

Hi list, We have balancer plugin in upmap mode running for a while now: health: HEALTH_OK pgs: 1973 active+clean 194 active+remapped+backfilling 73 active+remapped+backfill_wait recovery: 588 MiB/s, 343 objects/s Our objects are stored on EC pool. We got an PG_NOT_DEEP_SCRUBBED alert and have noticed that no scrubbing (literally zero) was done since the balancing started. Has anyone some ideas why this is happening? "pg deep-scrub <pgid>" did not help. Thanks! -- Vytenis

2 3

The sufficient OSD capabilities to enable write access on cephfs
by Derrick Lin 29 May '20

29 May '20

Hi guys, I have a Ceph Cluster up and running and cephfs created (all done by ceph-ansible). I following the guide to mount the volume on CentOS7 via FUSE. When I mount the volume as the default admin (client.admin), everything works fine just like normal file system. Then I created a new client just for FUSE mount purpose, follow this guide: https://docs.ceph.com/docs/master/cephfs/mount-prerequisites/ The ceph fs authorize command created a new client with the following caps: [client.wp_test] key = AQDAEc9ebLXjGhAAxEGqTuTvCOoN30g4UzF5jw== caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rw tag cephfs data=cephfs" It can mount the volume, and I can touch a file. But when I tried write data, such as editing a new text or cat a file, I got some read-only error or [root@mon-6-26 ceph_root]# cat test.txt cat: test.txt: Operation not permitted if I modified the OSD cap to "allow *", the it allows write again. Can anyone suggest what have been done incorrectly? We are using 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) Cheers, Derrick

2 2

PGs degraded after osd restart
by Chad William Seys 29 May '20

29 May '20

You may have run into this bug: https://tracker.ceph.com/issues/44286

2 1

Repo for Nautilus packages for CentOS8
by Massimo Sgaravatto 29 May '20

29 May '20

Hi What is the repo supposed to be used for Nautilus packages for CentOS8 ? The documentation says to use https://download.ceph.com/rpm-nautilus, but https://download.ceph.com/rpm-nautilus/el8 is empty I see some packages in: http://mirror.centos.org/centos/8/storage/x86_64/ceph-nautilus/Packages/c/ but the latest rpms published there are for the 14.2.7 release (I can't find 14.2.9 packages) Thanks, Massimo

1 0

Recover UUID from a partition
by Szabo, Istvan (Agoda) 29 May '20

29 May '20

Hi, Is there a way to recover UUID from the partition? Someone mapped in fstab to /sd* not UUID and all the metadata is gone. The data is there , just can't mount it to access the data. Any idea how to get it back or determine what was it before? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Ceph Tech Talk: What's New In Octopus
by Mike Perez 29 May '20

29 May '20

Hi everyone, We had a Ceph Tech Talk on May 28th at 17:00 UTC from Josh Durgin and Lenz Grimmer on a summary of the new features and enhancements with the new Octopus release. You can watch the recording here: https://www.youtube.com/watch?list=PLrBUGiINAakM36YJiTT0qYepZTVncFDdc&v=UGU… You can see our archive here: https://ceph.io/ceph-tech-talks/ We have live streams of Ceph Tech Talks every last Thursday of each month at 17:00 UTC. If you're interested in giving a talk next month, please reply to me directly! -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 0

15.2.2 bluestore issue
by Josh Durgin 28 May '20

28 May '20

Hi folks, at this time we recommend pausing OSD upgrades to 15.2.2. There have been a couple reports of OSDs crashing due to rocksdb corruption after upgrading to 15.2.2 [1] [2]. It's safe to upgrade monitors and mgr, but OSDs and everything else should wait. We're investigating and will get a fix out as soon as we can. You can follow progress on this tracker: https://tracker.ceph.com/issues/45613 Josh [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CX5PRFGL6UB… [2] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CWN7BNPGSRB…

3 4

PGs degraded after osd restart
by Vytenis A 28 May '20

28 May '20

Hi cephists, We have a 10 node cluster running Nautilus 14.2.9 All objects are on EC pool. We have mgr balancer plugin in upmap mode doing it's rebalancing: health: HEALTH_OK pgs: 1985 active+clean 190 active+remapped+backfilling 65 active+remapped+backfill_wait io: client: 0 B/s wr, 0 op/s rd, 0 op/s wr recovery: 770 MiB/s, 463 objects/s We have restarted osd.0 on one of our OSD nodes, and this was the status immediately after: ``` health: HEALTH_WARN 1 osds down Degraded data redundancy: 4531479/531067647 objects degraded (0.853%), 109 pgs degraded ``` Then OSD became UP again: ``` health: HEALTH_WARN Degraded data redundancy: 4963207/531067545 objects degraded (0.935%), 120 pgs degraded ``` And after a minute or so has passed it settled on: ``` health: HEALTH_WARN Degraded data redundancy: 295515/531067347 objects degraded (0.056%), 10 pgs degraded, 10 pgs undersized ``` upmap balancer was running during osd.0 restart, the restart was successfull, without any issues. This left us wondering - how could a simple osd restart cause degraded PGs? Could this be related to the upmap balancer running? Thanks! -- Vyteni

1 0

snapshot-based mirroring explanation in docs
by Hans van den Bogert 28 May '20

28 May '20

Hi list, When reading the documentation for the new way of mirroring [1], some questions arose, especially with the following sentence: > Since this mode is not point-in-time consistent, the full snapshot delta will need to be synced prior to use during a failover scenario. 1) I'm not sure I follow. Why are snapshots not point-in-time, whereas using the journal-based mirroring is? 2) Also, I am not sure what the implications are for the second part of the referenced sentence, what significance does the word "full" bear in "full snapshot delta" ? Further, Isn't it obvious that in order to do a failover, the other side should have the latest snapshot? - I am probably missing what the documentation is trying to tell me. All in all, I think some background concepts are missing in order to read this chapter of the documentation. It might well be that this is explained somewhere else in the docs, but maybe we should link to that. If someone can explain the above 2 points, I'm willing to take a shot at making a patch if I can make this clearer in the docs. Thanks, Hans [1] https://docs.ceph.com/docs/octopus/rbd/rbd-mirroring/

2 1

cephfs file layouts, empty objects in first data pool
by Håkan T Johansson 28 May '20

28 May '20

Hi, running 14.2.6, debian buster (backports). Have set up a cephfs with 3 data pools and one metadata pool: myfs_data, myfs_data_hdd, myfs_data_ssd, and myfs_metadata. The data of all files are with the use of ceph.dir.layout.pool either stored in the pools myfs_data_hdd or myfs_data_ssd. This has also been checked by dumping the ceph.file.layout.pool attributes of all files. The filesystem has 1617949 files and 36042 directories. There are however approximately as many objects in the first pool created for the cephfs, myfs_data, as there are files. They also becomes more or fewer as files are created or deleted (so cannot be some leftover from earlier exercises). Note how the USED size is reported as 0 bytes, correctly reflecting that no file data is stored in them. POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR myfs_data 0 B 1618229 0 4854687 0 0 0 2263590 129 GiB 23312479 124 GiB 0 B 0 B myfs_data_hdd 831 GiB 136309 0 408927 0 0 0 106046 200 GiB 269084 277 GiB 0 B 0 B myfs_data_ssd 43 GiB 1552412 0 4657236 0 0 0 181468 2.3 GiB 4661935 12 GiB 0 B 0 B myfs_metadata 1.2 GiB 36096 0 108288 0 0 0 4828623 82 GiB 1355102 143 GiB 0 B 0 B Is this expected? I was assuming that in this scenario, all objects, both their data and any keys would be either in the metadata pool, or the two pools where the objects are stored. Is it some additional metadata keys that are stored in the first created data pool for cephfs? This would not be so nice in case the osd selection rules for it are using worse disks than the data itself... Btw: is there any tool to see the amount of key value data size associated with a pool? 'ceph osd df' gives omap and meta for osds, but not broken down per pool. Best regards, Håkan

4 6

Re: Reducing RAM usage on production MDS
by Dan van der Ster 28 May '20

28 May '20

Hi Dylan, It looks like you have 10GB of heap to be release -- try `ceph tell mds.$(hostname) heap release` to free that up. Otherwise, I've found it safe to incrementally inject decreased mds_cache_memory_limit's on prod mds's running v12.2.12. I'd start by decreasing the size just a few hundred MBs at a time while tailing the mds log with `debug mds = 2` or `watch --color ceph fs status` to see the cache sizes decrease and stabilize after each change. (In my case I've decreased from ~16GB caches to ~4GB across 9 active MDSs -- I moved at around 500MB per injection and there were no slow req's or client issues.) BTW, we also increase `mds cache trim threshold` to allow the MDS to trim more caps per 5s tick -- if you find the lru is not trimming quickly enough you could try 1.5 or 2x the default value. If things get hairy you could increase the mds_beacon_grace (on mon and or mds) to tolerate longer missed heartbeats rather than failing the mds. Cheers, Dan On Thu, May 28, 2020 at 7:09 AM Dylan McCulloch <dmc(a)unimelb.edu.au> wrote: > > Hi all, > > The single active MDS on one of our Ceph clusters is close to running out of RAM. > > MDS total system RAM = 528GB > MDS current free system RAM = 4GB > mds_cache_memory_limit = 451GB > current mds cache usage = 426GB > > Presumably we need to reduce our mds_cache_memory_limit and/or mds_max_caps_per_client, but would like some guidance on whether it’s possible to do that safely on a live production cluster when the MDS is already pretty close to running out of RAM. > > Cluster is Luminous - 12.2.12 > Running single active MDS with two standby. > 890 clients > Mix of kernel client (4.19.86) and ceph-fuse. > Clients are 12.2.12 (398) and 12.2.13 (3) > > The kernel clients have stayed under “mds_max_caps_per_client”: “1048576". But the ceph-fuse clients appear to hold very large numbers according to the ceph-fuse asok. > e.g. > “num_caps”: 1007144398, > “num_caps”: 1150184586, > “num_caps”: 1502231153, > “num_caps”: 1714655840, > “num_caps”: 2022826512, > > Dropping caches on the clients appears to reduce their cap usage but does not free up RAM on the MDS. > What is the safest method to free cache and reduce RAM usage on the MDS in this situation (without having to evict or remount clients)? > I’m concerned that reducing mds_cache_memory_limit even in very small increments may trigger a large recall of caps and overwhelm the MDS. > We also considered setting a reduced mds_cache_memory_limit on both the standby MDS. Would a subsequent failover to an MDS with a lower cache limit be safe? > Some more details below and I’d be more than happy to provide additional logs. > > Thanks, > Dylan > > > # free -b > total used free shared buff/cache available > Mem: 540954992640 535268749312 4924698624 438284288 761544704 3893182464 > Swap: 0 0 0 > > # ceph daemon mds.$(hostname -s) config get mds_cache_memory_limit > { > "mds_cache_memory_limit": "450971566080" > } > > # ceph daemon mds.$(hostname -s) cache status > { > "pool": { > "items": 10593257843, > "bytes": 425176150288 > } > } > > # ceph daemon mds.$(hostname -s) dump_mempools | grep -A2 "mds_co\|anon" > "buffer_anon": { > "items": 3935, > "bytes": 4537932 > -- > "mds_co": { > "items": 10595391186, > "bytes": 425255456209 > > # ceph daemon mds.$(hostname -s) perf dump | jq '.mds_mem.rss' > 520100552 > > # ceph tell mds.$(hostname) heap stats > tcmalloc heap stats:------------------------------------------------ > MALLOC: 496040753720 (473061.3 MiB) Bytes in use by application > MALLOC: + 11085479936 (10571.9 MiB) Bytes in page heap freelist > MALLOC: + 22568895888 (21523.4 MiB) Bytes in central cache freelist > MALLOC: + 31744 ( 0.0 MiB) Bytes in transfer cache freelist > MALLOC: + 34186296 ( 32.6 MiB) Bytes in thread cache freelists > MALLOC: + 2802057216 ( 2672.2 MiB) Bytes in malloc metadata > MALLOC: ------------ > MALLOC: = 532531404800 (507861.5 MiB) Actual memory used (physical + swap) > MALLOC: + 1315700736 ( 1254.8 MiB) Bytes released to OS (aka unmapped) > MALLOC: ------------ > MALLOC: = 533847105536 (509116.3 MiB) Virtual address space used > MALLOC: > MALLOC: 44496459 Spans in use > MALLOC: 22 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > ------------------------------------------------ > Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). > Bytes released to the OS take up virtual address space but no physical memory. > > > # ceph fs status > hpc_projects - 890 clients > ============ > +------+--------+----------------+---------------+-------+-------+ > | Rank | State | MDS | Activity | dns | inos | > +------+--------+----------------+---------------+-------+-------+ > | 0 | active | mds1-ceph2-qh2 | Reqs: 304 /s | 167M | 167M | > +------+--------+----------------+---------------+-------+-------+ > +--------------------+----------+-------+-------+ > | Pool | type | used | avail | > +--------------------+----------+-------+-------+ > | hpcfs_metadata | metadata | 17.4G | 1893G | > | hpcfs_data | data | 1014T | 379T | > | test_nvmemeta | data | 0 | 1893G | > | hpcfs_data_sandisk | data | 312T | 184T | > +--------------------+----------+-------+-------+ > > +----------------+ > | Standby MDS | > +----------------+ > | mds3-ceph2-qh2 | > | mds2-ceph2-qh2 | > +----------------+ > MDS version: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Reducing RAM usage on production MDS
by Dylan McCulloch 28 May '20

28 May '20

Hi all, The single active MDS on one of our Ceph clusters is close to running out of RAM. MDS total system RAM = 528GB MDS current free system RAM = 4GB mds_cache_memory_limit = 451GB current mds cache usage = 426GB Presumably we need to reduce our mds_cache_memory_limit and/or mds_max_caps_per_client, but would like some guidance on whether it’s possible to do that safely on a live production cluster when the MDS is already pretty close to running out of RAM. Cluster is Luminous - 12.2.12 Running single active MDS with two standby. 890 clients Mix of kernel client (4.19.86) and ceph-fuse. Clients are 12.2.12 (398) and 12.2.13 (3) The kernel clients have stayed under “mds_max_caps_per_client”: “1048576". But the ceph-fuse clients appear to hold very large numbers according to the ceph-fuse asok. e.g. “num_caps”: 1007144398, “num_caps”: 1150184586, “num_caps”: 1502231153, “num_caps”: 1714655840, “num_caps”: 2022826512, Dropping caches on the clients appears to reduce their cap usage but does not free up RAM on the MDS. What is the safest method to free cache and reduce RAM usage on the MDS in this situation (without having to evict or remount clients)? I’m concerned that reducing mds_cache_memory_limit even in very small increments may trigger a large recall of caps and overwhelm the MDS. We also considered setting a reduced mds_cache_memory_limit on both the standby MDS. Would a subsequent failover to an MDS with a lower cache limit be safe? Some more details below and I’d be more than happy to provide additional logs. Thanks, Dylan # free -b total used free shared buff/cache available Mem: 540954992640 535268749312 4924698624 438284288 761544704 3893182464 Swap: 0 0 0 # ceph daemon mds.$(hostname -s) config get mds_cache_memory_limit { "mds_cache_memory_limit": "450971566080" } # ceph daemon mds.$(hostname -s) cache status { "pool": { "items": 10593257843, "bytes": 425176150288 } } # ceph daemon mds.$(hostname -s) dump_mempools | grep -A2 "mds_co\|anon" "buffer_anon": { "items": 3935, "bytes": 4537932 -- "mds_co": { "items": 10595391186, "bytes": 425255456209 # ceph daemon mds.$(hostname -s) perf dump | jq '.mds_mem.rss' 520100552 # ceph tell mds.$(hostname) heap stats tcmalloc heap stats:------------------------------------------------ MALLOC: 496040753720 (473061.3 MiB) Bytes in use by application MALLOC: + 11085479936 (10571.9 MiB) Bytes in page heap freelist MALLOC: + 22568895888 (21523.4 MiB) Bytes in central cache freelist MALLOC: + 31744 ( 0.0 MiB) Bytes in transfer cache freelist MALLOC: + 34186296 ( 32.6 MiB) Bytes in thread cache freelists MALLOC: + 2802057216 ( 2672.2 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 532531404800 (507861.5 MiB) Actual memory used (physical + swap) MALLOC: + 1315700736 ( 1254.8 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 533847105536 (509116.3 MiB) Virtual address space used MALLOC: MALLOC: 44496459 Spans in use MALLOC: 22 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. # ceph fs status hpc_projects - 890 clients ============ +------+--------+----------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+----------------+---------------+-------+-------+ | 0 | active | mds1-ceph2-qh2 | Reqs: 304 /s | 167M | 167M | +------+--------+----------------+---------------+-------+-------+ +--------------------+----------+-------+-------+ | Pool | type | used | avail | +--------------------+----------+-------+-------+ | hpcfs_metadata | metadata | 17.4G | 1893G | | hpcfs_data | data | 1014T | 379T | | test_nvmemeta | data | 0 | 1893G | | hpcfs_data_sandisk | data | 312T | 184T | +--------------------+----------+-------+-------+ +----------------+ | Standby MDS | +----------------+ | mds3-ceph2-qh2 | | mds2-ceph2-qh2 | +----------------+ MDS version: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)

1 0

Nautilus to Octopus Upgrade mds without downtime
by Andreas Schiefer 28 May '20

28 May '20

Hello, if I understand correctly: if we upgrade from an running nautilus cluster to octopus we have a downtime on an update of MDS. Is this correct? Mit freundlichen Grüßen / Kind regards Andreas Schiefer Leiter Systemadministration / Head of systemadministration --- HOME OF LOYALTY CRM- & Customer Loyalty Solution by UW Service Gesellschaft für Direktwerbung und Marketingberatung mbH Alter Deutzer Postweg 221 51107 Koeln (Rath/Heumar) Deutschland Telefon : +49 221 98696 0 Telefax : +49 221 98696 5222 info(a)uw-service.de www.hooloy.de Amtsgericht Koeln HRB 24 768 UST-ID: DE 164 191 706 Geschäftsführer: Ralf Heim ---

2 1

Fwd: [IO-500] IO500 ISC20 Call for Submission
by John Bent 27 May '20

27 May '20

FYI. Hope to see some awesome CephFS submissions for our virtual IO500 BoF! Thanks, John ---------- Forwarded message --------- From: committee--- via IO-500 <io-500(a)vi4io.org> Date: Fri, May 22, 2020 at 1:53 PM Subject: [IO-500] IO500 ISC20 Call for Submission To: <io-500(a)vi4io.org> *Deadline*: 08 June 2020 AoE The IO500 <http://io500.org/> is now accepting and encouraging submissions for the upcoming 6th IO500 list. Once again, we are also accepting submissions to the 10 Node Challenge to encourage the submission of small scale results. The new ranked lists will be announced via live-stream at a virtual session. We hope to see many new results. The benchmark suite is designed to be easy to run and the community has multiple active support channels to help with any questions. Please note that submissions of all sizes are welcome; the site has customizable sorting so it is possible to submit on a small system and still get a very good per-client score for example. Additionally, the list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. More details below. Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 <http://io500.org/> was created in 2017, published its first list at SC17, and has grown exponentially since then. The need for such an initiative has long been known within High-Performance Computing; however, defining appropriate benchmarks had long been challenging. Despite this challenge, the community, after long and spirited discussion, finally reached consensus on a suite of benchmarks and a metric for resolving the scores into a single ranking. The multi-fold goals of the benchmark suite are as follows: 1. Maximizing simplicity in running the benchmark suite 2. Encouraging optimization and documentation of tuning parameters for performance 3. Allowing submitters to highlight their “hero run” performance numbers 4. Forcing submitters to simultaneously report performance for challenging IO patterns. Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly constrained parameters forcing a difficult usage pattern in an attempt to determine a lower-bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Submitters are encouraged to share their tuning insights for publication. The goals of the community are also multi-fold: 1. Gather historical data for the sake of analysis and to aid predictions of storage futures 2. Collect tuning data to share valuable performance optimizations across the community 3. Encourage vendors and designers to optimize for workloads beyond “hero runs” 4. Establish bounded expectations for users, procurers, and administrators *10 Node I/O Challenge* The 10 Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly *10 client nodes* must be used to run the benchmark. You may use any shared storage with, e.g., any number of servers. When submitting for the IO500 list, you can opt-in for “Participate in the 10 compute node challenge only”, then we will not include the results into the ranked list. Other 10-node node submissions will be included in the full list and in the ranked list. We will announce the result in a separate derived list and in the full list but not on the ranked IO500 list at https://io500.org/. This information and rules for ISC20 submissions are available here: https://www.vi4io.org/io500/rules/submission Thanks, The IO500 Committee _______________________________________________ IO-500 mailing list IO-500(a)vi4io.org https://www.vi4io.org/mailman/listinfo/io-500

1 0

Cannot repair inconsistent PG
by Daniel Aberger - Profihost AG 27 May '20

27 May '20

Hello, we are currently experiencing problems with ceph pg repair not working on Ceph Nautilus 14.2.8. ceph health detail is showing us an inconsistent pg: [aaaaax-yyyy ~]# ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 18.19a is active+clean+inconsistent+snaptrim_wait, acting [21,15,39,18,0,9] when we try to repair it, nothing happens. [aaaaax-yyyy ~]# ceph pg repair 18.19a instructing pg 18.19as0 on osd.21 to repair There are no new entries in OSD 21's log file. We have no trouble repairing pgs in our other clusters so I assume it might have to be something related to this cluster using Erasure Codings. But this is just a wild guess. I found a similar problem in this mailing list - http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-April/026304.html Unfortunately the solution of waiting more than a week until it fixes itself isn't quite satisfying. Is there anyone who has had similar issues and knows how to repair these inconsistent pgs or what is causing the delay? -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

3 4

Luminous, OSDs down: "osd init failed" and "failed to load OSD map for epoch ... got 0 bytes"
by Fulvio Galeazzi 27 May '20

27 May '20

Hallo all, hope you can help me with very strange problems which arose suddenly today. Tried to search, also in this mailing list, but could not find anything relevant. At some point today, without any action from my side, I noticed some OSDs in my production cluster would go down and never come up. I am on Luminous 12.2.13, CentOS7, kernel 3.10: my setup is non-standard as OSD disks are served off a SAN (which is for sure OK now, although I cannot exclude some glitch). Tried to reboot OSD servers a few times, ran "activate --all", added bluestore_ignore_data_csum=true in the [osd] section in ceph.conf... the number of "down" OSDs changed for a while but now seems rather stable. There are actually two classes of problems (bit more details right below): - ERROR: osd init failed: (5) Input/output error - failed to load OSD map for epoch 141282, got 0 bytes *First problem* This affects 50 OSDs (all disks of this kind, on all but one server): these OSDs are reserved for object storage but I am not yet using them so I may in principle recreate them. But would be interested in understanding what the problem is, and learn how to solve it for future reference. Here is what I see in logs: ..... 2020-05-21 21:17:48.661348 7fa2e9a95ec0 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/cephpa1-72/block size 14.5TiB 2020-05-21 21:17:48.661428 7fa2e9a95ec0 1 bluefs mount 2020-05-21 21:17:48.662040 7fa2e9a95ec0 1 bluefs _init_alloc id 1 alloc_size 0x10000 size 0xe83a3400000 2020-05-21 21:52:43.858464 7fa2e9a95ec0 -1 bluefs mount failed to replay log: (5) Input/output error 2020-05-21 21:52:43.858589 7fa2e9a95ec0 1 fbmap_alloc 0x55c6bba92e00 shutdown 2020-05-21 21:52:43.858728 7fa2e9a95ec0 -1 bluestore(/var/lib/ceph/osd/cephpa1-72) _open_db failed bluefs mount: (5) Input/output error 2020-05-21 21:52:43.858790 7fa2e9a95ec0 1 bdev(0x55c6bbdb6600 /var/lib/ceph/osd/cephpa1-72/block) close 2020-05-21 21:52:44.103536 7fa2e9a95ec0 1 bdev(0x55c6bbdb8600 /var/lib/ceph/osd/cephpa1-72/block) close 2020-05-21 21:52:44.352899 7fa2e9a95ec0 -1 osd.72 0 OSD:init: unable to mount object store 2020-05-21 21:52:44.352956 7fa2e9a95ec0 -1 ESC[0;31m ** ERROR: osd init failed: (5) Input/output errorESC[0m *Second problem* This affects 11 OSDs, which I use *in production* for Cinder block storage: looks like all PGs for this pool are currently OK. Here is the excerpt from the logs. ..... -5> 2020-05-21 20:52:06.756469 7fd2ccc19ec0 0 _get_class not permitted to load kvs -4> 2020-05-21 20:52:06.759686 7fd2ccc19ec0 1 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/cls/rgw/cls_rgw.cc:3869: Loaded rgw class! -3> 2020-05-21 20:52:06.760021 7fd2ccc19ec0 1 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/cls/log/cls_log.cc:299: Loaded log class! -2> 2020-05-21 20:52:06.760730 7fd2ccc19ec0 1 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/cls/replica_log/cls_replica_log.cc:135: Loaded replica log class! -1> 2020-05-21 20:52:06.760873 7fd2ccc19ec0 -1 osd.63 0 failed to load OSD map for epoch 141282, got 0 bytes 0> 2020-05-21 20:52:06.763277 7fd2ccc19ec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fd2ccc19ec0 time 2020-05-21 20:52:06.760916 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/osd/OSD.h: 994: FAILED assert(ret) Has anyone any idea how I could fix these problems, or what I could do to try and shed some light? And also, what caused them, and whether there is some magic configuration flag I could use to protect my cluster? Thanks a lot for your help! Fulvio

2 5

High latency spikes under jewel
by Bence Szabo 27 May '20

27 May '20

Hi, We experienced random and relative high latency spikes (around 0.5-10 sec) in our ceph cluster which consists 6 osd nodes, all osd nodes have 6 osd-s. One osd built with one spinning disk and two nvme device. We use a bcache device for osd back end (mixed with hdd and an nvme partition as caching device) and one nvme partition for journal. This synthetic command can be use for check io and latency: rados bench -p rbd 10 write -b 4000 -t 64 With this parameters we often got about 1.5 sec or higher for maximum latency. We cannot decide if our cluster is misconfigured or just this is a natural ceph behavior. Any help, suggestion would be appreciated. Regards, Bence -- --Szabo Bence --<szabo.bence(a)gmail.com>

2 1

osds dropping out of the cluster w/ "OSD::osd_op_tp thread … had timed out"
by thoralf schulze 27 May '20

27 May '20

hi there, we are seeing osd occasionally getting kicked out of our cluster, after having been marked down by other osds. most of the time, the affected osd rejoins the cluster after about ~5 minutes, but sometimes this takes much longer. during that time, the osd seems to run just fine. this happens more often that we'd like it to … is "OSD::osd_op_tp thread … had timed out" a real error condition or just a warning about certain operations on the osd taking a long time? i already set osd_op_thread_timeout to 120 (was 60 before, default should be 15 according to the docs), but apparently that doesn't make any difference. are there any other settings that prevent this kind of behaviour? mon_osd_report_timeout maybe, as in frank schilder's case? the cluster runs nautilus 14.2.7, osds are backed by spinning platters with their rocksdb and wals on nvmes. in general, there seems to be the following pattern: - it happens under moderate to heavy load, eg. while creating pools with a lot of pgs - the affected osd logs a lot of: "heartbeat_map is_healthy 'OSD::osd_op_tp thread ${thread-id}' had timed out after 60" … and finally something along the lines of: May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.211 7fb25cc80700 0 bluestore(/var/lib/ceph/osd/ceph-293) log_latency_fn slow operation observed for _collection_list, latency = 96.337s, lat = 96s cid =2.0s2_head start GHMAX end GHMAX max 30 May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.219 7fb25cc80700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fb25cc80700' had timed out after 60 May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: osd.293 osd.293 2 : Monitor daemon marked osd.293 down, but it is still running May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.315 7fb267c96700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.293 down, but it is still running May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.315 7fb267c96700 0 log_channel(cluster) do_log log to syslog May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.315 7fb267c96700 0 log_channel(cluster) log [DBG] : map e646639 wrongly marked me down at e646638 May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.315 7fb267c96700 0 log_channel(cluster) do_log log to syslog May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.371 7fb272cac700 -1 osd.293 646639 set_numa_affinity unable to identify public interface 'br-bond0' numa node: (2) No such file or directory May 18 21:12:34 ceph-osd-05 ceph-osd[2356578]: 2020-05-18 21:12:34.371 7fb272cac700 -1 osd.293 646639 set_numa_affinity unable to identify public interface 'br-bond0' numa node: (2) No such file or directory - meanwhile on the mon: 2020-05-18 21:12:16.440 7f08f7933700 0 mon.ceph-mon-01@0(leader) e4 handle_command mon_command({"prefix": "status"} v 0) v1 entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2020-05-18 21:12:18.436 7f08f7933700 0 log_channel(cluster) log [DBG] : osd.293 reported failed by osd.101 2020-05-18 21:12:18.848 7f08f7933700 0 log_channel(cluster) log [DBG] : osd.293 reported failed by osd.533 [… lots of these from various osds] 2020-05-18 21:12:24.992 7f08f7933700 0 log_channel(cluster) log [DBG] : osd.293 reported failed by osd.421 2020-05-18 21:12:26.124 7f08f7933700 0 log_channel(cluster) log [DBG] : osd.293 reported failed by osd.504 2020-05-18 21:12:26.132 7f08f7933700 0 log_channel(cluster) log [INF] : osd.293 failed (root=tuberlin,datacenter=barz,host=ceph-osd-05) (16 reporters from different host after 27.137527 >= grace 26.361774) 2020-05-18 21:12:26.236 7f08fa138700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN) 2020-05-18 21:12:26.280 7f08f6130700 0 log_channel(cluster) log [DBG] : osdmap e646638: 604 total, 603 up, 604 in 2020-05-18 21:12:27.336 7f08f6130700 0 log_channel(cluster) log [DBG] : osdmap e646639: 604 total, 603 up, 604 in 2020-05-18 21:12:28.248 7f08fa138700 0 log_channel(cluster) log [WRN] : Health check failed: Reduced data availability: 17 pgs peering (PG_AVAILABILITY) 2020-05-18 21:12:29.392 7f08fa138700 0 log_channel(cluster) log [WRN] : Health check failed: Degraded data redundancy: 80091/181232010 objects degraded (0.044%), 18 pgs degraded (PG_DEGRADED) 2020-05-18 21:12:33.927 7f08fa138700 0 log_channel(cluster) log [INF] : Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 22 pgs peering) 2020-05-18 21:12:35.095 7f08fa138700 0 log_channel(cluster) log [INF] : Health check cleared: OSD_DOWN (was: 1 osds down) 2020-05-18 21:12:35.119 7f08f6130700 0 log_channel(cluster) log [INF] : osd.293 [v2:172.28.9.26:6936/2356578,v1:172.28.9.26:6937/2356578] boot 2020-05-18 21:12:35.119 7f08f6130700 0 log_channel(cluster) log [DBG] : osdmap e646640: 604 total, 604 up, 604 in 2020-05-18 21:12:36.175 7f08f6130700 0 log_channel(cluster) log [DBG] : osdmap e646641: 604 total, 604 up, 604 in i can happily provide more detailed logs, if that helps. thank you very much & with kind regards, thoralf.

3 7

Re: [External Email] Re: Ceph Nautius not working after setting MTU 9000
by Paul Emmerich 27 May '20

27 May '20

Don't optimize stuff without benchmarking *before and after*, don't apply random tuning tipps from the Internet without benchmarking them. My experience with Jumbo frames: 3% performance. On a NVMe-only setup with 100 Gbit/s network. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, May 26, 2020 at 7:02 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Look what I have found!!! :) > https://ceph.com/geen-categorie/ceph-loves-jumbo-frames/ > > > > -----Original Message----- > From: Anthony D'Atri [mailto:anthony.datri@gmail.com] > Sent: maandag 25 mei 2020 22:12 > To: Marc Roos > Cc: kdhall; martin.verges; sstkadu; amudhan83; ceph-users; doustar > Subject: Re: [ceph-users] Re: [External Email] Re: Ceph Nautius not > working after setting MTU 9000 > > Quick and easy depends on your network infrastructure. Sometimes it is > difficult or impossible to retrofit a live cluster without disruption. > > > > On May 25, 2020, at 1:03 AM, Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > > > I am interested. I am always setting mtu to 9000. To be honest I > > cannot imagine there is no optimization since you have less interrupt > > requests, and you are able x times as much data. Every time there > > something written about optimizing the first thing mention is changing > > > to the mtu 9000. Because it is quick and easy win. > > > > > > > > > > -----Original Message----- > > From: Dave Hall [mailto:kdhall@binghamton.edu] > > Sent: maandag 25 mei 2020 5:11 > > To: Martin Verges; Suresh Rama > > Cc: Amudhan P; Khodayar Doustar; ceph-users > > Subject: [ceph-users] Re: [External Email] Re: Ceph Nautius not > > working after setting MTU 9000 > > > > All, > > > > Regarding Martin's observations about Jumbo Frames.... > > > > I have recently been gathering some notes from various internet > > sources regarding Linux network performance, and Linux performance in > > general, to be applied to a Ceph cluster I manage but also to the rest > > > of the Linux server farm I'm responsible for. > > > > In short, enabling Jumbo Frames without also tuning a number of other > > kernel and NIC attributes will not provide the performance increases > > we'd like to see. I have not yet had a chance to go through the rest > > of the testing I'd like to do, but I can confirm (via iperf3) that > > only enabling Jumbo Frames didn't make a significant difference. > > > > Some of the other attributes I'm referring to are incoming and > > outgoing buffer sizes at the NIC, IP, and TCP levels, interrupt > > coalescing, NIC offload functions that should or shouldn't be turned > > on, packet queuing disciplines (tc), the best choice of TCP slow-start > > > algorithms, and other TCP features and attributes. > > > > The most off-beat item I saw was something about adding IPTABLES rules > > > to bypass CONNTRACK table lookups. > > > > In order to do anything meaningful to assess the effect of all of > > these settings I'd like to figure out how to set them all via Ansible > > - so more to learn before I can give opinions. > > > > --> If anybody has added this type of configuration to Ceph Ansible, > > I'd be glad for some pointers. > > > > I have started to compile a document containing my notes. It's rough, > > > but I'd be glad to share if anybody is interested. > > > > -Dave > > > > Dave Hall > > Binghamton University > > > >> On 5/24/2020 12:29 PM, Martin Verges wrote: > >> > >> Just save yourself the trouble. You won't have any real benefit from > > MTU > >> 9000. It has some smallish, but it is not worth the effort, problems, > > and > >> loss of reliability for most environments. > >> Try it yourself and do some benchmarks, especially with your regular > >> workload on the cluster (not the maximum peak performance), then drop > > the > >> MTU to default ;). > >> > >> Please if anyone has other real world benchmarks showing huge > > differences > >> in regular Ceph clusters, please feel free to post it here. > >> > >> -- > >> Martin Verges > >> Managing director > >> > >> Mobile: +49 174 9335695 > >> E-Mail: martin.verges(a)croit.io > >> Chat: https://t.me/MartinVerges > >> > >> croit GmbH, Freseniusstr. 31h, 81247 Munich > >> CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht > >> Munich HRB 231263 > >> > >> Web: https://croit.io > >> YouTube: https://goo.gl/PGE1Bx > >> > >> > >>> Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama > >> <sstkadu(a)gmail.com>: > >> > >>> Ping with 9000 MTU won't get response as I said and it should be > > 8972. Glad > >>> it is working but you should know what happened to avoid this issue > > later. > >>> > >>>> On Sun, May 24, 2020, 3:04 AM Amudhan P <amudhan83(a)gmail.com> > wrote: > >>> > >>>> No, ping with MTU size 9000 didn't work. > >>>> > >>>> On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar > > <doustar(a)rayanexon.ir> > >>>> wrote: > >>>> > >>>>> Does your ping work or not? > >>>>> > >>>>> > >>>>> On Sun, May 24, 2020 at 6:53 AM Amudhan P <amudhan83(a)gmail.com> > > wrote: > >>>>> > >>>>>> Yes, I have set setting on the switch side also. > >>>>>> > >>>>>> On Sat 23 May, 2020, 6:47 PM Khodayar Doustar, > > <doustar(a)rayanexon.ir> > >>>>>> wrote: > >>>>>> > >>>>>>> Problem should be with network. When you change MTU it should be > >>>> changed > >>>>>>> all over the network, any single hup on your network should > >>>>>>> speak > > and > >>>>>>> accept 9000 MTU packets. you can check it on your hosts with > >>> "ifconfig" > >>>>>>> command and there is also equivalent commands for other > >>>> network/security > >>>>>>> devices. > >>>>>>> > >>>>>>> If you have just one node which it not correctly configured for > > MTU > >>>> 9000 > >>>>>>> it wouldn't work. > >>>>>>> > >>>>>>> On Sat, May 23, 2020 at 2:30 PM sinan(a)turka.nl <sinan(a)turka.nl> > >>> wrote: > >>>>>>>> Can the servers/nodes ping eachother using large packet sizes? > >>>>>>>> I > >>> guess > >>>>>>>> not. > >>>>>>>> > >>>>>>>> Sinan Polat > >>>>>>>> > >>>>>>>>> Op 23 mei 2020 om 14:21 heeft Amudhan P <amudhan83(a)gmail.com> > > het > >>>>>>>> volgende geschreven: > >>>>>>>>> In OSD logs "heartbeat_check: no reply from OSD" > >>>>>>>>> > >>>>>>>>>> On Sat, May 23, 2020 at 5:44 PM Amudhan P > > <amudhan83(a)gmail.com> > >>>>>>>> wrote: > >>>>>>>>>> Hi, > >>>>>>>>>> > >>>>>>>>>> I have set Network switch with MTU size 9000 and also in my > >>> netplan > >>>>>>>>>> configuration. > >>>>>>>>>> > >>>>>>>>>> What else needs to be checked? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Sat, May 23, 2020 at 3:39 PM Wido den Hollander < > >>> wido(a)42on.com > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> On 5/23/20 12:02 PM, Amudhan P wrote: > >>>>>>>>>>>> Hi, > >>>>>>>>>>>> > >>>>>>>>>>>> I am using ceph Nautilus in Ubuntu 18.04 working fine wit > > MTU > >>>> size > >>>>>>>> 1500 > >>>>>>>>>>>> (default) recently i tried to update MTU size to 9000. > >>>>>>>>>>>> After setting Jumbo frame running ceph -s is timing out. > >>>>>>>>>>> Ceph can run just fine with an MTU of 9000. But there is > >>> probably > >>>>>>>>>>> something else wrong on the network which is causing this. > >>>>>>>>>>> > >>>>>>>>>>> Check the Jumbo Frames settings on all the switches as well > > to > >>>> make > >>>>>>>> sure > >>>>>>>>>>> they forward all the packets. > >>>>>>>>>>> > >>>>>>>>>>> This is definitely not a Ceph issue. > >>>>>>>>>>> > >>>>>>>>>>> Wido > >>>>>>>>>>> > >>>>>>>>>>>> regards > >>>>>>>>>>>> Amudhan P > >>>>>>>>>>>> _______________________________________________ > >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To > >>>>>>>>>>>> unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>>>>>>>>>> > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe > > >>>>>>>>>>> send an email to ceph-users-leave(a)ceph.io > >>>>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe > >>>>>>>>> send an email to ceph-users-leave(a)ceph.io > >>>>>>>> _______________________________________________ > >>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe > >>>>>>>> send an email to ceph-users-leave(a)ceph.io > >>>>>>>> > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send > >>>> an email to ceph-users-leave(a)ceph.io > >>>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > >>> email to ceph-users-leave(a)ceph.io > >>> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > >> email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

4 4

looking for telegram group in English or Chinese
by Zhenshi Zhou 27 May '20

27 May '20

Hi all, Is there any telegram group for communicating with ceph users?

3 3

Multisite RADOS Gateway replication factor in zonegroup
by alexander.vysochin＠Megafon.ru 27 May '20

27 May '20

Hello, I didn’t find any information about the replication factor in the zone group. Assume I have three ceph clusters with Rados Gateway in one zone group each with replica size 3. How many replicas of an object I’ll get in total? Is it possible to define several regions, each with several datacenters, and define maximum replication factor at region scope? Александр Высочин Старший разработчик Технические инновации и инфраструктура - Разработка и развертывание платформ бизнес-сервисов Технические инновации и инфраструктура ПАО «МегаФон» <tel:> [cid:image001.png@01D632C0.86CD4F00] ________________________________ Информация в этом сообщении предназначена исключительно для конкретных лиц, которым она адресована. В сообщении может содержаться конфиденциальная информация, которая не может быть раскрыта или использована кем-либо кроме адресатов. Если вы не адресат этого сообщения, то использование, переадресация, копирование или распространение содержания сообщения или его части - незаконно и запрещено. Если Вы получили это сообщение ошибочно, пожалуйста, незамедлительно сообщите отправителю об этом и удалите со всем содержимым само сообщение и любые возможные его копии и приложения. Настоящее сообщение и вложения в него носят исключительно информационный характер и не влекут для нас каких-либо обязательств, не означают признания или подтверждения нами каких-либо обстоятельств. Такие последствия наступают только после подписания уполномоченными лицами оригинальных версий соглашений, актов, уведомлений или иных документов. В случае неприемлемости для нас каких-либо условий мы сохраняем за собой право в любое время прекратить переговоры в отношении любых вопросов. Вступая с нами в переписку, Вы считаетесь проинформированными обо всем, что указано выше. The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. The contents may not be disclosed or used by anyone other than the addressee. If you are not the intended recipient(s), any use, disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this communication in error please notify us immediately by responding to this email and then delete the e-mail and all attachments and any copies thereof. This communication and attachments hereto are for the informational purposes only and do not create or modify any our obligations and shall not be deemed as our admission or confirmation of any circumstances. Such consequences may occur only after duly authorized persons have signed the originals of the agreements, acts or other documents. If any conditions are unacceptable for us we reserve a right to terminate negotiations in respect of any issues at any time. Entering into any correspondence with us you are considered to be informed on all that is stated above. -----

2 1

Prometheus Python Errors
by support＠orbitingcode.com 26 May '20

26 May '20

Hello Everyone, I have installed both Prometheus and Grafana on one of my manager nodes (Ubuntu 18.04), and have configured both according to the documentation. I have visible Grafana dashboards when visiting http://mon1:3000, but no data exists on the dashboard. Python errors are shown for the job_name: ceph in Prometheus. Below is my prometheus.yaml configuration global: scrape_interval: 5s scrape_configs: - job_name: prometheus static_configs: - targets: ['localhost:9090'] - job_name: 'ceph-exporter' static_configs: - targets: ['localhost:9100'] labels: alias: ceph-exporter - job_name: 'ceph' static_configs: - targets: ['localhost:9283'] labels: alias: ceph And, these are the Python errors shown when I view the details of the targets in Prometheus (http://mon1:9090/targets) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 670, in respond response.body = self.handler() File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 220, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 60, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1060, in metrics return self._metrics(instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1074, in _metrics instance.collect_cache = instance.collect() File "/usr/share/ceph/mgr/prometheus/module.py", line 975, in collect self.get_rbd_stats() File "/usr/share/ceph/mgr/prometheus/module.py", line 734, in get_rbd_stats 'rbd_stats_pools_refresh_interval', 300) TypeError: unsupported operand type(s) for +: 'int' and 'str' If anyone has experienced this issue, and might have a solution, I would appreciate any assistance. Thank you, Todd

2 1

Ceph Nautius not working after setting MTU 9000
by Amudhan P 26 May '20

26 May '20

Hi, I am using ceph Nautilus in Ubuntu 18.04 working fine wit MTU size 1500 (default) recently i tried to update MTU size to 9000. After setting Jumbo frame running ceph -s is timing out. regards Amudhan P

10 20

move bluestore wal/db
by Frank R 26 May '20

26 May '20

Is there a safe way to move the bluestore wal and db to a new device that doesn't involve rebuilding the entire OSD?

2 1

Ceph client on rhel6?
by Simon Sutter 26 May '20

26 May '20

Hello again, I have a new question: We want to upgrade a server, with an os based on rhel6. The ceph cluster is atm on octopus. How can I install the client packages to mount cephfs and do a backup of the server? Is it even possible? Are the client packages from hammer compatible with the octopus release? Thanks in advance, Simon

1 0

mds container dies during deployment
by Simon Sutter 26 May '20

26 May '20

Hello everyone I've got a fresh ceph octopus installation and I'm trying to set up a cephfs with erasure code configuration. The metadata pool was set up as default. The erasure code pool was set up with this command: -> ceph osd pool create ec-data_fs 128 erasure default Enabled overwrites: -> ceph osd pool set ec-data_fs allow_ec_overwrites true And create fs: -> ceph fs new ec-data_fs meta_fs ec-data_fs --force Then I tried deploying the mds, but this fails: -> ceph orch daemon add mds ec-data_fs magma01 returns: -> Deployed mds.ec-data_fs.magma01.ujpcly on host 'magma01' The mds daemon is not there. Aparently the container dies without any information, as seen in the journal: May 25 16:11:56 magma01 podman[9348]: 2020-05-25 16:11:56.670510456 +0200 CEST m=+0.186462913 container create 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:56 magma01 systemd[1]: Started libpod-conmon-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope. May 25 16:11:56 magma01 systemd[1]: Started libcontainer container 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90. May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.112182262 +0200 CEST m=+0.628134873 container init 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137011897 +0200 CEST m=+0.652964354 container start 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137110412 +0200 CEST m=+0.653062853 container attach 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 systemd[1]: libpod-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope: Consumed 327ms CPU time May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.182968802 +0200 CEST m=+0.698921275 container died 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.413743787 +0200 CEST m=+0.929696266 container remove 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) Can someone help me debugging this? Cheers Simon

1 1

Nautilus: (Minority of) OSDs with huge buffer_anon usage - triggering OOMkiller in worst cases.
by aoanla＠gmail.com 26 May '20

26 May '20

Hi, Following on from various woes, we see an odd and unhelpful behaviour with some OSDs on our cluster currently. A minority of OSDs seem to have runaway memory usage, rising to 10s of GB, whilst other OSDs on the same host behave sensibly. This started when we moved from Mimic -> Nautilus, as far as we can tell. In best case, this causes some nodes to start swapping [and reduces their performance]. In worst case, it triggers the OOMkiller. I have dumped the mempool for these OSDs, which shows that almost all the memory is in the buffer_anon pool. The perf dump shows that the OSD is targetting the 4GB limit that's set for it, but for some reason is unable to reach this due to stuff in the priorty_cache (which seems to be mostly what is filling buffer_anon) Can anyone advise on what we should do next? (mempool dump and excerpt of perf dump at end of email). Thanks for any help, Sam Skipsey MEMPOOL DUMP { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 5629372, "bytes": 45034976 }, "bluestore_cache_data": { "items": 127, "bytes": 65675264 }, "bluestore_cache_onode": { "items": 8275, "bytes": 4634000 }, "bluestore_cache_other": { "items": 2967913, "bytes": 62469216 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 145, "bytes": 100920 }, "bluestore_writing_deferred": { "items": 335, "bytes": 13160884 }, "bluestore_writing": { "items": 1406, "bytes": 5379120 }, "bluefs": { "items": 1105, "bytes": 24376 }, "buffer_anon": { "items": 13705143, "bytes": 40719040439 }, "buffer_meta": { "items": 6820143, "bytes": 600172584 }, "osd": { "items": 96, "bytes": 1138176 }, "osd_mapbl": { "items": 59, "bytes": 7022524 }, "osd_pglog": { "items": 491049, "bytes": 156701043 }, "osdmap": { "items": 107885, "bytes": 1723616 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 29733053, "bytes": 41682277138 } } } PERF DUMP excerpt: "prioritycache": { "target_bytes": 4294967296, "mapped_bytes": 38466584576, "unmapped_bytes": 425984, "heap_bytes": 38467010560, "cache_bytes": 134217728 },

2 4

Performance issues in newly deployed Ceph cluster
by Loschwitz,Martin Gerhard 26 May '20

26 May '20

Folks, I am running into a very strange issue with a brand new Ceph cluster during initial testing. Cluster consists of 12 nodes, 4 of them have SSDs only, the other eight have a mixture of SSDs and HDDs. The latter nods are configured so that three or four HDDs use one SSDs for their blockdb. Ceph version is Nautilus. When writing to the cluster, clients will, in regular intervals, run into I/O stall (i.e. writes will take up to 25 minutes to complete). Deleting RBD Images will often take forever as well. After several weeks of debugging, what I can say from looking at the log files, is that what appears to take a lot of time is writing stuff to OSDs: "time": "2020-05-20 10:52:23.211006", "event": "reached_pg" }, { "time": "2020-05-20 10:52:23.211047", "event": "waiting for ondisk" }, { "time": "2020-05-20 10:53:35.369081", "event": "done" } But these machines are I/O idling. there is almost no I/O happening at all according to sysstat. I am slowly growing a bit desperate over this, and hence I wonder whether anybody has ever seen a similar issue? Or are there possibly any tips on where to carry on with debugging? Servers are from Dell with PERC controllers in HBA mode. The primary purpose of this Ceph cluster is to serve as backing storage for OpenStack, and to this point, I was not able to reproduce the issue with the SSD-only nodes. Best regards Martin

1 0

RGW Multisite metadata sync
by Sailaja Yedugundla 26 May '20

26 May '20

Hi, I am trying to setup multisite cluster with 2 sites. I created master zonegroup and zone by following the instructions given in the documentation. On the secondary zone cluster I could pull the master zone. I created secondary zone. When I tried to commit the period I am getting the following error. 2020-05-25 16:16:46.054 7f4ad25596c0 1 Cannot find zone id=2f272093-3712-45a7-8a63-b17f12ccd07c (name=testsite2), switching to local zonegroup configuration Sending period to new master zone 6d8d5ffa-2034-4717-978e-3ab4ba4349c5 request failed: (5) Input/output error failed to commit period: (5) Input/output error Can someone please help me to solve this issue. Regards, Sailaja

2 3

Mismatched object counts between "rados df" and "rados ls" after rbd images removal
by icy chan 26 May '20

26 May '20

Hi, The numbers of object counts from "rados df" and "rados ls" are different in my testing environment. I think it maybe some zero bytes or unclean objects since I removed all rbd images on top of it few days ago. How can I make it right / found out where are those ghost objects? Or i should ignore it since the numbers was not that high. $ rados -p rbd df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR rbd 18 MiB 430107 0 1290321 0 0 0 141243877 6.9 TiB 42395431 11 TiB 0 B 0 B $ rados -p rbd ls | wc -l 4 $ rados -p rbd ls gateway.conf rbd_directory rbd_info rbd_trash Regs, Icy

2 5

RGW Multi-site Issue
by Zhenshi Zhou 26 May '20

26 May '20

Hi, I am new on rgw and try deploying a mutisite cluster in order to sync data from one cluster to another. My source zone is the default zone in the default zonegroup, structure as belows: realm: big-realm | zonegroup: default / \ master zone: default secondary zone: backup *STEP*: on source cluster: 1. radosgw-admin realm create --rgw-realm=big-realm --default 2. radosgw-admin zonegroup modify --rgw-realm big-realm --rgw-zonegroup default --master --endpoints "http://172.24.29.26:7480" 3. radosgw-admin zone modify --rgw-zonegroup default --rgw-zone default --master --endpoints "http://172.24.29.26:7480" 4. radosgw-admin user create --uid=sync-user --display-name="Synchronization User" --access-key=redhat --secret=redhat --system 5. radosgw-admin zone modify --rgw-zone=default --access-key=redhat --secret=redhat 6. radosgw-admin period update --commit on destination cluster: 1. radosgw-admin realm pull --url="http://172.24.29.26:7480 <http://172.24.29.26/>" --access-key=redhat --secret=redhat --rgw-realm=big-realm 2. radosgw-admin realm default --rgw-realm=big-realm 3. radosgw-admin period pull --url="http://172.24.29.26:7480" --access-key=redhat --secret=redhat 4. radosgw-admin zonegroup default --rgw-zonegroup=default 5. radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=backup --endpoints="http://172.24.29.29:7480" --access-key=redhat --secret=redhat --default 6. radosgw-admin period update --commit commit period on secondary zone get error: 2020-04-02 14:36:04.707 7fd8ee9376c0 1 Cannot find zone id=8c75360a-c0cf-4772-b85e-ff74726396c2 (name=backup), switching to local zonegroup configuration Sending period to new master zone 5fba7cae-47f1-4c8e-9a34-1b499c9c27f8 request failed: (2202) Unknown error 2202 failed to commit period: (2202) Unknown error 2202 radosgw-admin sync status: 2020-04-02 14:37:18.330 7f27c60676c0 1 Cannot find zone id=8c75360a-c0cf-4772-b85e-ff74726396c2 (name=backup), switching to local zonegroup configuration realm fec73799-36be-4418-abb2-9804cc83d83d (big-realm) zonegroup fc61ac2f-dc1d-421b-90af-ffe9113b9935 (default) zone 8c75360a-c0cf-4772-b85e-ff74726396c2 (backup) metadata sync failed to read sync status: (2) No such file or directory data sync source: 5fba7cae-47f1-4c8e-9a34-1b499c9c27f8 (default) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source My source cluster's version is 13.2.8, and destination cluster's is 14.2.8. I tried sync data from cluster of both 13.2.8 version and got the same error. Is there any step I was wrong or the default zone cannot be synced? Thanks

3 4

Cephfs IO halt on Node failure
by Amudhan P 25 May '20

25 May '20

Hi, I am using ceph Nautilus cluster with below configuration. 3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr are running in shared mode. The client mounted through ceph kernel client. I was trying to emulate a node failure when a write and read were going on (replica2) pool. I was expecting read and write continue after a small pause due to a Node failure but it halts and never resumes until the failed node is up. I remember I tested the same scenario before in ceph mimic where it continued IO after a small pause. regards Amudhan P

3 8

May Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 25 May '20

25 May '20

Hello, We will be having a Ceph science/research/big cluster call on Wednesday May 27th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200527 We try to keep it to an hour or less. Ceph calendar event details: May 27, 2020 14:00 UTC 4pm Central European 9am Central US (I think I got timezones right this time) Description:Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index <https://www.google.com/url?q=https://pad.ceph.com/p/Ceph_Science_User_Group…> Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink <https://www.google.com/url?q=https://bluejeans.com/908675367?src%3Dcalendar…> To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc <https://www.google.com/url?q=http://bjn.vc&sa=D&ust=1579363980705000&usg=AO…> 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers <https://www.google.com/url?q=https://www.redhat.com/en/conference-numbers&s…> 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 <https://www.google.com/url?q=https://bluejeans.com/111&sa=D&ust=15793639807…> Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 0

RGW resharding
by Adrian Nicolae 25 May '20

25 May '20

Hi, I have the following Ceph Mimic setup : - a bunch of old servers with 3-4 SATA drives each (74 OSDs in total) - index/leveldb is stored on each OSD (so no SSD drives, just SATA) - the current usage is : GLOBAL: SIZE AVAIL RAW USED %RAW USED 542 TiB 105 TiB 437 TiB 80.67 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS .rgw.root 1 1.1 KiB 0 26 TiB 4 default.rgw.control 2 0 B 0 26 TiB 8 default.rgw.meta 3 20 MiB 0 26 TiB 75357 default.rgw.log 4 0 B 0 26 TiB 4271 default.rgw.buckets.data 5 290 TiB 85.05 51 TiB 78067284 default.rgw.buckets.non-ec 6 0 B 0 26 TiB 0 default.rgw.buckets.index 7 0 B 0 26 TiB 603008 - rgw_override_bucket_index_max_shards = 16. Clients are accessing RGW via Swift, not S3. - the replication schema is EC 4+2. We are using this Ceph cluster as a secondary storage for another storage infrastructure (which is more expensive) and we are offloading cold data (big files with a low number of downloads/reads from our customer). This way we can lower the TCO . So most of the files are big ( a few GB at least). So far Ceph is doing well considering that I don't have big expectations from current hardware. I'm a bit worried however that we have 78 M objects with max_shards=16 and we will probably reach 100M in the next few months. Do I need a increase the max shards to ensure the stability of the cluster ? I read that storing more than 1 M of objects in a single bucket can lead to OSD's flapping or having io timeouts during deep-scrub or even to have ODS's failures due to the leveldb compacting all the time if we have a large number of DELETEs. Any advice would be appreciated. Thank you, Adrian Nicolae

2 4

Disable auto-creation of RGW pools
by Katarzyna Myrek 25 May '20

25 May '20

Hi We have some clusters which are rbd only. Each time someone uses radosgw-admin by mistake on those clusters, rgw pools are auto created. Is there a way to disable that? I mean the part: "When radosgw first tries to operate on a zone pool that does not exist, it will create that pool with the default values from osd pool default pg num and osd pool default pgp num" Thanks, Kate

1 0

Issue adding mon after upgrade to 15.2.2
by Erwin Bogaard 25 May '20

25 May '20

In our lab-setup, I'm simulating the future migration of centos 7 + ceph 14.2.x to cepntos 8 + ceph 15.2.x. At the moment, I upgraded one of the nodes, which is a combined mon+mgr+mds+osd, to el8 + 15.2.2. The other node (also a combined one) is still on el7 + 14.2.9. The osd was detected and re-added easily, as were the mgr and mds. The mon won't add. I use the following (manual) method: 1. ceph auth get mon. -o /tmp/kr (exports correctly) 2. ceph mon getmap -o /tmp/mm (exports correctly) 3. sudo ceph-mon -i ontw-ceph01 --mkfs --monmap /tmp/mm --keyring /tmp/kr (creates file system + rocksdb correctly in "/var/lib/ceph/mon/ceph-ontw-ceph01") 4. systemctl start ceph-mon(a)ontw-ceph01.service ceph-mon starts up successfully, but won't enter the cluster. In the other (14.2.9) ceph-mon log, I see loads of this message (multiple per : -------------- 2020-05-25 13:07:38.075 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3 adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of hints 2020-05-25 13:07:38.075 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3 adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of hints 2020-05-25 13:07:38.076 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3 adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of hints 2020-05-25 13:07:38.076 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3 adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of hints 2020-05-25 13:07:38.077 7fb38366d700 1 mon.ontw-ceph02@0(leader) e3 adding peer [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] to list of hints etc... -------------- The 3300 and 6789 ports are open and reacheable on both nodes and they can reach and connect to each other. 192.168.100.60 = ontw-ceph01, upgraded 15.2.2-node 192.168.100.61 = ontw-ceph02, active 14.2.9-node When I start ceph-mon in debug mode @ ceph01, I see the following happening: -------------- 2020-05-25T13:07:35.031+0200 7f712e1d46c0 0 starting mon.ontw-ceph01 rank -1 at public addrs [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] at bind addrs [v2:192.168.100.60:3300/0,v1:192.168.100.60:6789/0] mon_data /var/lib/ceph/mon/ceph-ontw-ceph01 fsid f3c3f099-2940-4074-a7fe-1aea6259f67b 2020-05-25T13:07:35.033+0200 7f712e1d46c0 1 mon.ontw-ceph01@-1(???) e1 preinit fsid f3c3f099-2940-4074-a7fe-1aea6259f67b 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 check_fsid cluster_uuid contains 'f3c3f099-2940-4074-a7fe-1aea6259f67b' 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?)} 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 calc_quorum_requirements required_features 2449958197560098820 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 required_features 2449958197560098820 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 has_ever_joined = 0 2020-05-25T13:07:35.033+0200 7f712e1d46c0 1 mon.ontw-ceph01@-1(???) e1 initial_members ontw-ceph02,ontw-ceph01, filtering seed monmap 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 monmap is e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2: 192.168.100.61:3300/0,v1:192.168.100.61:6789/0]} 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 extra probe peers 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 sync_last_committed_floor 0 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 init_paxos 2020-05-25T13:07:35.033+0200 7f712e1d46c0 5 mon.ontw-ceph01(a)-1(???).mds e0 Unable to load 'last_metadata' 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).health init 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).config init 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 refresh_from_paxos 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 refresh_from_paxos no cluster_fingerprint 2020-05-25T13:07:35.033+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).log v0 update_from_paxos 2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).log v0 update_from_paxos version 0 summary v 0 2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).auth v0 update_from_paxos 2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).config load_config got 0 keys 2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).mgrstat 0 2020-05-25T13:07:35.034+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).mgrstat check_subs 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).health update_from_paxos 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(???) e1 loading initial keyring to bootstrap authentication for mkfs 2020-05-25T13:07:35.035+0200 7f712e1d46c0 2 mon.ontw-ceph01@-1(???) e1 init 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(???).mgr e0 prime_mgr_client 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 bootstrap 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 sync_reset_requester 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 unregister_cluster_logger - not registered 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 monmap e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2: 192.168.100.61:3300/0,v1:192.168.100.61:6789/0]} 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 _reset 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01(a)-1(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 timecheck_finish 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 scrub_event_cancel 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 scrub_reset 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 reset_probe_timeout 0x563148b1eb60 after 2 seconds 2020-05-25T13:07:35.035+0200 7f712e1d46c0 10 mon.ontw-ceph01@-1(probing) e1 probing other monitors 2020-05-25T13:07:35.035+0200 7f712e1d46c0 0 -- [v2: 192.168.100.60:3300/0,v1:192.168.100.60:6789/0] send_to message mon_probe(probe f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph01 new mon_release octopus) v7 with empty dest 2020-05-25T13:07:35.036+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1 get_authorizer for mon 2020-05-25T13:07:35.238+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1 get_authorizer for mon 2020-05-25T13:07:35.640+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1 get_authorizer for mon 2020-05-25T13:07:36.442+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1 get_authorizer for mon 2020-05-25T13:07:37.035+0200 7f711a308700 4 mon.ontw-ceph01@-1(probing) e1 probe_timeout 0x563148b1eb60 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 bootstrap 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 sync_reset_requester 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 unregister_cluster_logger - not registered 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 monmap e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2: 192.168.100.61:3300/0,v1:192.168.100.61:6789/0]} 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 _reset 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01(a)-1(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 timecheck_finish 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 scrub_event_cancel 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 scrub_reset 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 reset_probe_timeout 0x563148b1f7c0 after 2 seconds 2020-05-25T13:07:37.035+0200 7f711a308700 10 mon.ontw-ceph01@-1(probing) e1 probing other monitors 2020-05-25T13:07:37.035+0200 7f711a308700 0 -- [v2: 192.168.100.60:3300/0,v1:192.168.100.60:6789/0] send_to message mon_probe(probe f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph01 new mon_release octopus) v7 with empty dest 2020-05-25T13:07:38.045+0200 7f711bb0b700 10 mon.ontw-ceph01@-1(probing) e1 get_authorizer for mon 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1 _ms_dispatch new session 0x563149893680 MonSession(mon.0 [v2: 192.168.100.61:3300/0,v1:192.168.100.61:6789/0] is open , features 0x3f01cfb8ffacffff (luminous)) features 0x3f01cfb8ffacffff 2020-05-25T13:07:38.047+0200 7f7117b03700 5 mon.ontw-ceph01@-1(probing) e1 _ms_dispatch setting monitor caps on this connection 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1 handle_probe mon_probe(reply f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph02 quorum 0 paxos( fc 304966 lc 305658 ) mon_release nautilus) v7 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1 handle_probe_reply mon.0 v2:192.168.100.61:3300/0 mon_probe(reply f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph02 quorum 0 paxos( fc 304966 lc 305658 ) mon_release nautilus) v7 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1 monmap is e1: 2 mons at {ontw-ceph01=,ontw-ceph02=[v2: 192.168.100.61:3300/0,v1:192.168.100.61:6789/0]} 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e1 got newer/committed monmap epoch 3, mine was 1 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 bootstrap 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 sync_reset_requester 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 unregister_cluster_logger - not registered 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 cancel_probe_timeout 0x563148b1f7c0 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 monmap e3: 1 mons at {ontw-ceph02=[v2: 192.168.100.61:3300/0,v1:192.168.100.61:6789/0]} 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 _reset 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01(a)-1(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 timecheck_finish 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 scrub_event_cancel 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 scrub_reset 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 cancel_probe_timeout (none scheduled) 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 reset_probe_timeout 0x563148b1f7c0 after 2 seconds 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 probing other monitors 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 handle_probe mon_probe(reply f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph02 quorum 0 paxos( fc 304966 lc 305658 ) mon_release nautilus) v7 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 handle_probe_reply mon.0 v2:192.168.100.61:3300/0 mon_probe(reply f3c3f099-2940-4074-a7fe-1aea6259f67b name ontw-ceph02 quorum 0 paxos( fc 304966 lc 305658 ) mon_release nautilus) v7 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 monmap is e3: 1 mons at {ontw-ceph02=[v2: 192.168.100.61:3300/0,v1:192.168.100.61:6789/0]} 2020-05-25T13:07:38.047+0200 7f7117b03700 10 mon.ontw-ceph01@-1(probing) e3 got newer/committed monmap epoch 3, mine was 3 -------------- I'm a bit lost at what's (not) happening. Hope you can help!

1 0

Handling scrubbing/deep scrubbing
by Kamil Szczygieł 25 May '20

25 May '20

Hi, I've 4 node cluster with 13x15TB 7.2k OSDs each and around 300TB data inside. I'm having issues with deep scrub/scrub not being done in time, any tips to handle these operations with large disks like this? osd pool default size = 2 osd deep scrub interval = 2592000 osd scrub begin hour = 23 osd scrub end hour = 5 osd scrub sleep = 0.1 Cheers, Kamil

2 1

Re: [External Email] Re: Ceph Nautius not working after setting MTU 9000
by Janne Johansson 25 May '20

25 May '20

Den mån 25 maj 2020 kl 10:03 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > > I am interested. I am always setting mtu to 9000. To be honest I cannot > imagine there is no optimization since you have less interrupt requests, > and you are able x times as much data. Every time there something > written about optimizing the first thing mention is changing to the mtu > 9000. Because it is quick and easy win. > > This sort of assumes you are not using interrupt coalescing network cards, because if you do, you can get something like hundreds of packets in one single IRQ*, already checksummed and stripped and in recent cards (10-25-40GE) even delivered into the cpu L3 cache by the time you get the int, so if they were 1500 or 9000 on the wire doesn't matter much by then. Even in the bad old days of software handling of all parts packet-related, many things (like mbuf allocations) were optimized for 1500, so 9k packets became just a multiple of a number of 1500 bytes chunks taken from a pool of network buffers anyhow. I'm not trying to shoot down the 9k-vs-1500 idea, but doing a benchmark will give you lots more facts than airing things that are easy to imagine but really doesn't have a huge impact because hw manufacturers worked around things like this a long time ago. If your tests say you win x%, then use it by all means. I'm just not thinking that 10/25/40G networks are so filled that the frame overheads really matter as a matter of % of the packet sizes and the cards offload most of the work to strip the overhead out, so the computer won't notice it was ever there. *) SysKonnect cards had this around 2003, just to get a feeling for what "modern ethernet cards" means in this context. -- May the most significant bit of your life be positive.

1 0

No output from rbd perf image iotop/iostat
by Eugen Block 25 May '20

25 May '20

Hi all, I have a Nautilus cluster mostly used for RBD (openstack) and CephFS. I have been using rbd perf command from time to time but it doesn't work anymore. I have tried several images in different pools but there's no output at all except for client:~ $ rbd perf image iostat --format json volumes-ssd/volume-358cd6c5-6fb0-424f-93d9-990ea1963472 rbd: waiting for initial image stats It never updates, no matter how long I wait. It stopped working while we were using version 14.2.3, last Friday we updated to 14.2.9 but it still doesn't work. The only relevant mgr log output I'm seeing in debug mode (debug_mgr 5/5) is this: ---snip--- 2020-05-25 10:53:07.072 7fedd5f59700 4 mgr.server _handle_command decoded 4 2020-05-25 10:53:07.072 7fedd5f59700 4 mgr.server _handle_command prefix=rbd perf image stats 2020-05-25 10:53:07.072 7fedd5f59700 0 log_channel(audit) log [DBG] : from='client.710971242 v1:192.168.103.13:0/693257394' entity='client.admin' cmd=[ 2020-05-25 10:53:07.072 7fedd5f59700 0 log_channel(audit) log [DBG] : { 2020-05-25 10:53:07.072 7fedd5f59700 0 log_channel(audit) log [DBG] : "prefix": "rbd perf image stats", 2020-05-25 10:53:07.072 7fedd5f59700 0 log_channel(audit) log [DBG] : "pool_spec": "volumes-ssd/volume-358cd6c5-6fb0-424f-93d9-990ea1963472", 2020-05-25 10:53:07.072 7fedd5f59700 0 log_channel(audit) log [DBG] : "sort_by": "write_ops", 2020-05-25 10:53:07.072 7fedd5f59700 0 log_channel(audit) log [DBG] : "format": "json" 2020-05-25 10:53:07.072 7fedd5f59700 0 log_channel(audit) log [DBG] : }"]: dispatch 2020-05-25 10:53:07.072 7fedd675a700 4 mgr.server reply reply success 2020-05-25 10:53:07.104 7fedd5f59700 4 mgr.server handle_report from 0x555e60f31200 osd,33 2020-05-25 10:53:07.172 7fedd5f59700 4 mgr.server handle_report from 0x555e5c6c6d80 osd,15 2020-05-25 10:53:07.224 7fedf10c1700 4 mgr send_beacon active ---snip--- What I'm also wondering about is that the "format": "json" doesn't change even if I choose to run --format plain or xml. Does anyone experience the same? The missing output also applies to rbd perf image iotop. Any hints are appreciated. Regards, Eugen

1 0

remove secondary zone from multisite
by Zhenshi Zhou 25 May '20

25 May '20

Hi all, I'm gonna make my secondary zone offline. How to remove the secondary zone from a mutisite?

1 1

Re: RGW Garbage Collector
by Matt Benjamin 24 May '20

24 May '20

Hi Manuel, rgw_gc_obj_min_wait -- yes, this is how you control how long rgw waits before removing the stripes of deleted objects the following are more gc performance and proportion of available iops: rgw_gc_processor_max_time -- controls how long gc runs once scheduled; a large value might be 3600 rgw_gc_processor_period -- sets the gc cycle; smaller is more frequent If you want to make gc more aggressive when it is running, set the following (can be increased), which more than doubles the : rgw_gc_max_concurrent_io = 20 rgw_gc_max_trim_chunk = 32 If you want to increase gc fraction of total rgw i/o, increase these (mostly, concurrent_io). regards, Matt On Sun, May 24, 2020 at 4:02 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > Hi, > > Im looking for any experience optimizing garbage collector with the next configs: > > global advanced rgw_gc_obj_min_wait > global advanced rgw_gc_processor_max_time > global advanced rgw_gc_processor_period > > By default gc expire objects within 2 hours, we're looking to define expire in 10 minutes as our S3 cluster got heavy uploads and deletes. > > Are those params usable? For us doesn't have sense store delete objects 2 hours in a gc. > > Regards > Manuel > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

2 1

RGW Garbage Collector
by EDH - Manuel Rios 24 May '20

24 May '20

Hi, Im looking for any experience optimizing garbage collector with the next configs: global advanced rgw_gc_obj_min_wait global advanced rgw_gc_processor_max_time global advanced rgw_gc_processor_period By default gc expire objects within 2 hours, we're looking to define expire in 10 minutes as our S3 cluster got heavy uploads and deletes. Are those params usable? For us doesn't have sense store delete objects 2 hours in a gc. Regards Manuel

1 0

RGW REST API failed request with status code 403
by apely agamakou 24 May '20

24 May '20

Hi, Since my upgrade from 15.2.1 to 15.2.2 i've got this error message at the "Object Gateway" section of the dashboard. RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx000000000000000000017-005ecac06c' b'-e349-eu-west-1","HostId":"e349-eu-west-1-default"}') I did try to change my secret-key and access-key without success. I made a tcpdump, i didn't see any special thing like json escape character etc .. Somebody had the same issue ?? Regards.

1 0

Re: question on ceph node count
by tim taler 24 May '20

24 May '20

yep, my fault I meant replication = 3 .... > > but aren't PGs checksummed so from the remaining PG (given the > > checksum would be right) two new copies could be created? > > Assuming again 3R on 5 nodes, failure domain of host, if 2 nodes go down, there will be 1/3 copies available. Normally a 3R pool has min_size set to 2. > > You can set min_size to 1 temporarily, then those PGs will become active and copies will be created to restore redundancy, but if that remaining OSD is damaged, if there’s a DIMM flake, a cosmic ray, if the wrong OSD crashes or restarts at the wrong time, you can find yourself without the most recent copy of data and be unable to recover. It’s Russian Roulette. I see, but wouldn't ceph try to recreate redundancy by it's own (unless I'm explicitly tell it not to do so)? And if the I/O and load on the cluster isn't too high disk speed good net connectivity good it would recover fairly quickly into healthy redundancy state? Anyhow, I'm not planing on crashing two nodes ;-) I just wanted to get a feeling of how much more secure/robust a setup with five nodes compared to four nodes is.

1 0