March 2020 - ceph-users - lists.ceph.io

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

Ceph influxDB support versus Telegraf Ceph plugin?
by victorhooi＠yahoo.com 06 Aug '20

06 Aug '20

Hi, I've read that Ceph has some InfluxDB reporting capabilities inbuilt (https://docs.ceph.com/docs/master/mgr/influx/) However, Telegraf, which is the system reporting daemon for InfluxDB, also has a Ceph plugin (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph) Just curious what people's thoughts on the two are, or what they are using in production? Which is easier to deploy/maintain, have you found? Or more useful for alerting, or tracking performance gremlins? Thanks, Victor

2 2

IPv6 connectivity gone for Ceph Telemetry
by Wido den Hollander 24 Jul '20

24 Jul '20

Hi, I was just checking on a few (13) IPv6-only Ceph clusters and I noticed that they couldn't send their Telemetry data anymore: telemetry.ceph.com has address 8.43.84.137 This server used to have Dual-Stack connectivity while it was still hosted at OVH. It seemed to have moved to Red Hat, but lost IPv6 connectivity. How can we get this back? Wido

2 1

Problem with OSD::osd_op_tp thread had timed out and other connected issues
by Jan Pekař - Imatic 21 Jul '20

21 Jul '20

Hello, I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network Cluster was empty, fresh install. We filled cluster with data (small blocks) using RGW. Cluster is now used for testing so no client was using it during my admin operations mentioned below After a while (7TB of data / 40M objects uploaded) we decided, that we increase pg_num from 128 to 256 to better spread data and to speedup this operation, I've set ceph config set mgr target_max_misplaced_ratio 1 so that whole cluster rebalance as quickly as it can. I have 3 issues/questions below: 1) I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to restart with logged heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide timed out after 150 after a while OSD's were back so I continued after a while with my tests. My question - increasing number of PG with maximal target_max_misplaced_ratio was too much for that OSDs? It is not recommended to do it this way? I had no problem with this increase before, but configuration of cluster was slightly different and it was luminous version. 2) Rebuild was still slow so I increased number of backfills ceph tell osd.* injectargs "--osd-max-backfills 10" and reduced recovery sleep time ceph tell osd.* injectargs "--osd-recovery-sleep-hdd 0.01" and after few hours I noticed, that some of my OSD's were restarted during recovery, in log I can see ... |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 start_waiting_for_healthy | I observed network graph usage and network utilization was low during recovery (10Gbit was not saturated). So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that OSD is using threads and HDD timeouts are not influencing heartbeats to other OSD's and MON. It looks like it is not true. 3) After OSD was wrongly marked down I can see that cluster has object degraded. There were no degraded object before that. Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs degraded, 8 pgs undersized It means that this OSD disconnection causes data degraded? How is it possible, when no OSD was lost. Data should be on that OSD and after peering should be everything OK. With luminous I had no problem, after OSD up degraded objects where recovered/found during few seconds and cluster was healthy within seconds. Thank you very much for additional info. I can perform additional tests you recommend because cluster is used for testing purpose now. With regards Jan Pekar -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz | +420326555326 ============ --

4 7

bluestore_default_buffered_write = true
by Adam Koczarski 20 Jul '20

20 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

ceph qos
by 展荣臻（信泰） 26 Jun '20

26 Jun '20

Hi everyone: There are two types of qos in ceph(one based on tokenbucket algorithm,another based on mclock ). Which one I can use in nautilus production environment ？Thank you

2 1

ERROR: osd init failed: (1) Operation not permitted
by Ml Ml 22 Jun '20

22 Jun '20

Hello List, first of all: Yes - i made mistakes. Now i am trying to recover :-/ I had a healthy 3 node cluster which i wanted to convert to a single one. My goal was to reinstall a fresh 3 Node cluster and start with 2 nodes. I was able to healthy turn it from a 3 Node Cluster to a 2 Node cluster. Then the problems began. I started to change size=1 and min_size=1. Health was okay until here. Then over sudden both nodes got fenced...one node refused to boot, mons where missing, etc...to make long story short, here is where i am right now: root@node03:~ # ceph -s cluster b3be313f-d0ef-42d5-80c8-6b41380a47e3 health HEALTH_WARN 53 pgs stale 53 pgs stuck stale monmap e4: 2 mons at {0=10.15.15.3:6789/0,1=10.15.15.2:6789/0} election epoch 298, quorum 0,1 1,0 osdmap e6097: 14 osds: 9 up, 9 in pgmap v93644673: 512 pgs, 1 pools, 1193 GB data, 304 kobjects 1088 GB used, 32277 GB / 33366 GB avail 459 active+clean 53 stale+active+clean root@node03:~ # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 32.56990 root default -2 25.35992 host node03 0 3.57999 osd.0 up 1.00000 1.00000 5 3.62999 osd.5 up 1.00000 1.00000 6 3.62999 osd.6 up 1.00000 1.00000 7 3.62999 osd.7 up 1.00000 1.00000 8 3.62999 osd.8 up 1.00000 1.00000 19 3.62999 osd.19 up 1.00000 1.00000 20 3.62999 osd.20 up 1.00000 1.00000 -3 7.20998 host node02 3 3.62999 osd.3 up 1.00000 1.00000 4 3.57999 osd.4 up 1.00000 1.00000 1 0 osd.1 down 0 1.00000 9 0 osd.9 down 0 1.00000 10 0 osd.10 down 0 1.00000 17 0 osd.17 down 0 1.00000 18 0 osd.18 down 0 1.00000 my main mistakes seemd to be: -------------------------------- ceph osd out osd.1 ceph auth del osd.1 systemctl stop ceph-osd@1 ceph osd rm 1 umount /var/lib/ceph/osd/ceph-1 ceph osd crush remove osd.1 As far as i can tell, ceph waits and needs data from that OSD.1 (which i removed) root@node03:~ # ceph health detail HEALTH_WARN 53 pgs stale; 53 pgs stuck stale pg 0.1a6 is stuck stale for 5086.552795, current state stale+active+clean, last acting [1] pg 0.142 is stuck stale for 5086.552784, current state stale+active+clean, last acting [1] pg 0.1e is stuck stale for 5086.552820, current state stale+active+clean, last acting [1] pg 0.e0 is stuck stale for 5086.552855, current state stale+active+clean, last acting [1] pg 0.1d is stuck stale for 5086.552822, current state stale+active+clean, last acting [1] pg 0.13c is stuck stale for 5086.552791, current state stale+active+clean, last acting [1] [...] SNIP [...] pg 0.e9 is stuck stale for 5086.552955, current state stale+active+clean, last acting [1] pg 0.87 is stuck stale for 5086.552939, current state stale+active+clean, last acting [1] When i try to start ODS.1 manually, i get: -------------------------------------------- 2020-02-10 18:48:26.107444 7f9ce31dd880 0 ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 10210 2020-02-10 18:48:26.134417 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) backend xfs (magic 0x58465342) 2020-02-10 18:48:26.184202 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is supported and appears to work 2020-02-10 18:48:26.184209 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2020-02-10 18:48:26.184526 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2020-02-10 18:48:26.184585 7f9ce31dd880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_feature: extsize is disabled by conf 2020-02-10 18:48:26.309755 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2020-02-10 18:48:26.633926 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.642185 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.664273 7f9ce31dd880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2020-02-10 18:48:26.732154 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for clients 2020-02-10 18:48:26.732163 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400 was 8705, adjusting msgr requires for mons 2020-02-10 18:48:26.732167 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for osds 2020-02-10 18:48:26.732179 7f9ce31dd880 0 osd.1 6002 load_pgs 2020-02-10 18:48:31.939810 7f9ce31dd880 0 osd.1 6002 load_pgs opened 53 pgs 2020-02-10 18:48:31.940546 7f9ce31dd880 -1 osd.1 6002 log_to_monitors {default=true} 2020-02-10 18:48:31.942471 7f9ce31dd880 1 journal close /var/lib/ceph/osd/ceph-1/journal 2020-02-10 18:48:31.969205 7f9ce31dd880 -1 ESC[0;31m ** ERROR: osd init failed: (1) Operation not permittedESC[0m Its mounted: /dev/sdg1 3.7T 127G 3.6T 4% /var/lib/ceph/osd/ceph-1 Is there any way i can get the OSD.1 back in? Thanks a lot, mario

2 1

Re: Excessive write load on mons after upgrade from 12.2.13 -> 14.2.7
by Dan van der Ster 10 Jun '20

10 Jun '20

This means it has been applied: # ceph osd dump -f json | jq .require_osd_release "nautilus" -- dan On Mon, Feb 17, 2020 at 11:10 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > How do you check if you issued this command in the past? > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Excessive write load on mons after upgrade > from 12.2.13 -> 14.2.7 > > Hi Peter, > > could be a totally different problem but did you run the command "ceph > osd require-osd-release nautilus" after the upgrade? > We had poor performance after upgrading to nautilus and running this > command fixed it. The same was reported by others for previous updates. > Here is my original message regarding this issue: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/OYFRWSJXPV… > > We did not observe the master election problem though. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

Help! ceph-mon is blocked after shutting down and ip address changed
by occj＠qq.com 04 Jun '20

04 Jun '20

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) os :CentOS Linux release 7.7.1908 (Core) single node ceph cluster with 1 mon,1mgr,1 mds,1rgw and 12osds , but only cephfs is used. ceph -s is blocked after shutting down the machine (192.168.0.104), then ip address changed to 192.168.1.6 I created the monmap with monmap tool and update the ceph.conf , hosts file and then start ceph-mon. and the ceph-mon log: ... 2019-12-11 08:57:45.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1285.14s 2019-12-11 08:57:50.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1290.14s 2019-12-11 08:57:55.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1295.14s 2019-12-11 08:58:00.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1300.14s 2019-12-11 08:58:05.172 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1305.14s 2019-12-11 08:58:10.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1310.14s 2019-12-11 08:58:15.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1315.14s 2019-12-11 08:58:20.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1320.14s 2019-12-11 08:58:25.174 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1325.14s ... I changed IP back to 192.168.0.104 yeasterday, but all the same. # cat /etc/ceph/ceph.conf [client.libvirt] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor log file = /var/log/ceph/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor [client.rgw.ceph-node1.rgw0] host = ceph-node1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-node1.rgw0.log rgw frontends = beast endpoint=192.168.1.6:8080 rgw thread pool size = 512 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.1.0/24 fsid = e384e8e6-94d5-4812-bfbb-d1b0468bdef5 mon host = [v2:192.168.1.6:3300,v1:192.168.1.6:6789] mon initial members = ceph-node1 osd crush chooseleaf type = 0 osd pool default crush rule = -1 public network = 192.168.1.0/24 [osd] osd memory target = 7870655146

2 1

Radosgw PubSub Traffic
by Dustin Guerrero 02 Jun '20

02 Jun '20

Hey all, We’ve been running some benchmarks against Ceph which we deployed using the Rook operator in Kubernetes. Everything seemed to scale linearly until a point where I see a single OSD receiving much higher CPU load than the other OSDs (nearly 100% saturation). After some investigation we noticed a ton of pubsub traffic in the strace coming from the RGW pods like so: [pid 22561] sendmsg(77, {msg_name(0)=NULL, msg_iov(3)=[{"\21\2)\0\0\0\10\0:\1\0\0\10\0\0\0\0\0\10\0\0\0\0\0\0\20\0\0-\321\211K"..., 73}, {"\200\0\0\0pubsub.user.ceph-user-wwITOk"..., 314}, {"\0\303\34[\360\314\233\2138\377\377\377\377\377\377\377\377", 17}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL|MSG_MORE <unfinished …> I’ve checked other OSDs and only a single OSD receives these messages. I suspect its creating a bottleneck. Does anyone have an idea on why these are being generated or how to stop them? The pubsub sync module doesn’t appear to be enabled, and our benchmark is doing simple gets/puts/deletes. We’re running Ceph 14.2.5 nautilus Thank you!

2 2

OSD backups and recovery
by Ludek Navratil 30 May '20

30 May '20

HI all, what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local filesystem and backup the content using Amanda. I believe that there must a better way to do this but I couldn't find it in docs. I know that one option is to setup an archive zone, but it requires an additional ceph cluster that needs to be maintained and looked after. I would rather avoid that. How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me to recover the data correctly? Many thanks,Ludek

4 5

General question CephFS or RBD
by Willi Schiegel 29 May '20

29 May '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

2 1

Ceph and iSCSI
by Bobby 29 May '20

29 May '20

Hi all, I am new to Ceph. But I have a some good understanding of iSCSI protocol. I will dive into Ceph because it looks promising. I am particularly interested in Ceph-RBD. I have a request. Can you please tell me, if any, what are the common similarities between iSCSI and Ceph. If someone has to work on a common model for iSCSI and Ceph, what would be those significant points you would suggest to someone who has some understanding of iSCSI? Looking forward to answers. Thanks in advance :-) BR

2 1

Very bad performance on a ceph rbd pool via iSCSI to VMware esx
by Salsa 29 May '20

29 May '20

I have a 3 hosts, 10 4TB HDDs per host ceph storage set up. I deined a 3 replica rbd pool and some images and presented them to a Vmware host via ISCSI, but the write performance is so bad the I managed to freeze a VM doing a big rsync to a datastore inside ceph and had to reboot it's host (seems I've filled up Vmware's ISCSI queue). Right now I'm getting write latencies from 20ms to 80 ms (per OSD) and sometimes peaking at 600 ms (per OSD). Client throughput is giving me around 4 MBs. Using a 4MB stripe 1 image I got 1.955..359 B/s inside the VM. On a 1MB stripe 1 I got 2.323.206 B/s inside the same VM. I think the performance is way too slow, much more than should be and that I can fix this by correcting some configuration. Any advices? -- Salsa Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 2

Ceph and Windows - experiences or suggestions
by Lars Täuber 29 May '20

29 May '20

Hi there! I got the task to connect a Windows client to our existing ceph cluster. I'm looking for experiences or suggestions from the community. There came two possibilities to my mind: 1. iSCSI Target on RBD exported to Windows 2. NFS-Ganesha on CephFS exported to Windows Is there a third way exporting a ceph cluster to a windows machine? I have some experiences with CephFS. We have a small cluster successfully running for linux clients. I don't have experiences with RBD or iSCSI. The Windows machine will use the space for backups. The kind of data is unknown. I expect the data to be a MS-SQL dump and user data from a sharepoint system. The Windows admin does not care whether NFS or iSCSI is used. I'd be happy if some of you could share experiences. Thanks Lars

10 10

ceph with rdma can not mount with kernel
by 李亚锋 29 May '20

29 May '20

2 1

Ceph on CentOS 8?
by Jan Kasprzak 29 May '20

29 May '20

Hello, Ceph users, does anybody use Ceph on recently released CentOS 8? Apparently there are no el8 packages neither at download.ceph.com, nor in the native CentOS package tree. I am thinking about upgrading my cluster to C8 (because of other software running on it apart from Ceph). Do el7 packages simply work? Can they be rebuilt using rpmbuild --rebuild? Or is running Ceph on C8 more complicated than that? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | sir_clive> I hope you don't mind if I steal some of your ideas? laryross> As far as stealing... we call it sharing here. --from rcgroups

9 8

rbd image naming convention
by Palanisamy 29 May '20

29 May '20

Hello Team, We've integrated Ceph cluster storage with Kubernetes and provisioning volumes through rbd-provisioner. When we're creating volumes from yaml files in Kubernetes, pv > pvc > mounting to pod, In kubernetes end pvc are showing as meaningful naming convention as per yaml file defined. But in ceph cluster, rbd image name is creating with dynamic uid. During troubleshooting time, this will be tedious to find exact rbd image. Please find the provisioning logs in below pasted snippet. kubectl get pods,pv,pvc NAME READY STATUS RESTARTS AGE pod/sleepypod 1/1 Running 0 4m9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO Delete Bound default/test-dyn-pvc ceph-rbd 4m9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-dyn-pvc Bound pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO ceph-rbd 4m11s *rbd-provisioner logs* I1121 10:59:15.009012 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2" I1121 10:59:15.009092 1 controller.go:1087] provision "default/test-dyn-pvc" class "ceph-rbd": volume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" provisioned I1121 10:59:15.009138 1 controller.go:1101] provision "default/test-dyn-pvc" class "ceph-rbd": trying to save persistentvvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" I1121 10:59:15.020418 1 controller.go:1108] provision "default/test-dyn-pvc" class "ceph-rbd": persistentvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" saved I1121 10:59:15.020476 1 controller.go:1149] provision "default/test-dyn-pvc" class "ceph-rbd": succeeded I1121 10:59:15.020802 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-dyn-pvc", UID:"cd37d2d6-cecc-4a05-9736-c8d80abde7f5", APIVersion:"v1", ResourceVersion:"24545639", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 *rbd image details in Ceph cluster end* rbd -p kube ls --long NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2 1 GiB 2 is there way to setup proper naming convention for rbd image as well during kubernetes deployment itself. Kubernetes version: v1.15.5 Ceph cluster version: 14.2.2 nautilus (stable) *Best Regards,* *Palanisamy*

2 2

cephfs file layouts, empty objects in first data pool
by Håkan T Johansson 28 May '20

28 May '20

Hi, running 14.2.6, debian buster (backports). Have set up a cephfs with 3 data pools and one metadata pool: myfs_data, myfs_data_hdd, myfs_data_ssd, and myfs_metadata. The data of all files are with the use of ceph.dir.layout.pool either stored in the pools myfs_data_hdd or myfs_data_ssd. This has also been checked by dumping the ceph.file.layout.pool attributes of all files. The filesystem has 1617949 files and 36042 directories. There are however approximately as many objects in the first pool created for the cephfs, myfs_data, as there are files. They also becomes more or fewer as files are created or deleted (so cannot be some leftover from earlier exercises). Note how the USED size is reported as 0 bytes, correctly reflecting that no file data is stored in them. POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR myfs_data 0 B 1618229 0 4854687 0 0 0 2263590 129 GiB 23312479 124 GiB 0 B 0 B myfs_data_hdd 831 GiB 136309 0 408927 0 0 0 106046 200 GiB 269084 277 GiB 0 B 0 B myfs_data_ssd 43 GiB 1552412 0 4657236 0 0 0 181468 2.3 GiB 4661935 12 GiB 0 B 0 B myfs_metadata 1.2 GiB 36096 0 108288 0 0 0 4828623 82 GiB 1355102 143 GiB 0 B 0 B Is this expected? I was assuming that in this scenario, all objects, both their data and any keys would be either in the metadata pool, or the two pools where the objects are stored. Is it some additional metadata keys that are stored in the first created data pool for cephfs? This would not be so nice in case the osd selection rules for it are using worse disks than the data itself... Btw: is there any tool to see the amount of key value data size associated with a pool? 'ceph osd df' gives omap and meta for osds, but not broken down per pool. Best regards, Håkan

4 6

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

How many MDS servers
by Robert Ruge 07 May '20

07 May '20

Quick question Ceph guru's. For a 1.1PB raw cephfs system currently storing 191TB of data and 390 million objects (mostly small Python, ML training files etc.) how many MDS servers should I be running? System is Nautilus 14.2.8. I ask because up to know I have run one MDS with one standby-replay and occasionally it blows up with large memory consumption, 60Gb+ even though I have mds_cache_memory_limit = 32G and that was 16G until recently. It of course tries to restart on another MDS node fails again and after several attempts usually comes back up. Today I increased to two active MDS's but the question is what is the optimal number for a pretty active system? The single MDS seemed to regularly run around 1400 req/s and I often get up to six clients failing to respond to cache pressure. The current setup is: ceph fs status cephfs - 71 clients ====== +------+----------------+--------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+----------------+--------+---------------+-------+-------+ | 0 | active | a | Reqs: 447 /s | 12.0M | 11.9M | | 1 | active | b | Reqs: 154 /s | 1749k | 1686k | | 1-s | standby-replay | c | Evts: 136 /s | 1440k | 1423k | | 0-s | standby-replay | d | Evts: 402 /s | 16.8k | 298 | +------+----------------+--------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 160G | 169G | | cephfs_data | data | 574T | 140T | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ | w | | x | | y | | z | +-------------+ MDS version: ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) Regards. Robert Ruge Systems & Network Manager Faculty of Science, Engineering & Built Environment [cid:image001.png@01D36789.04BE09A0] Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone. Deakin University does not warrant that this email and any attachments are error or virus free.

4 4

Cluster blacklists MDS, can't start
by Robert LeBlanc 07 May '20

07 May '20

This is the second time this happened in a couple of weeks. The MDS locks up and the stand-by can't take over so the Montiors black list them. I try to unblack list them, but they still say this in the logs mds.0.1184394 waiting for osdmap 234947 (which blacklists prior instance) Looking at a pg dump, it looks like the epoch is passed that. $ ceph pg map 3.756 osdmap e234953 pg 3.756 (3.756) -> up [113,180,115] acting [113,180,115] Last time, it seemed to just recover after about an hour all by it's self. Any way to speed this up? Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 4

CephFS with active-active NFS Ganesha
by Michael Bisig 06 May '20

06 May '20

Hi all, I am trying to setup an active-active NFS Ganesha cluster (with two Ganeshas (v3.0) running in Docker containers). I could manage to get two Ganesha daemons running using the rados_cluster backend for active-active deployment. I have the grace db within the cephfs metadata pool in an own namespace which keeps track on the node status. Now, I can mount the exposed filesystem over NFS (v4.1, v4.2) with both daemons. So far so good. __ Testing high availability resulted in an unexpected behavior for that I am not sure whether it is intentional or whether it is a configuration problem. Problem: If both are running, no E or N flags are set within the grace db, as I expect. Once, one host goes down (or is taken down) ALL clients cannot read nor write to the mounted filesystem, even the clients which are not connected to dead ganesha. In the db, I see that the dead ganesha has state NE and the active has E. This state is what I expect from the Ganesha documentation. Nevertheless, I would assume that the clients connected to the active daemon are not blocked. This state is not cleaned up by itself (e.g. after the grace period). I can unlock this situation by 'lifting' the dead node with a direct db call (using ganesha-rados-grace tool). But within an active-active deployment this is not suitable. The ganesha config looks like: ------------ NFS_CORE_PARAM { Enable_NLM = false; Protocols = 4; } NFSv4 { RecoveryBackend = rados_cluster; Minor_Versions = 1,2; } RADOS_KV { pool = "cephfsmetadata"; nodeid = "a" ; namespace = "grace"; UserId = "ganesha"; Ceph_Conf = "/etc/ceph/ceph.conf"; } MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; } EXPORT { Export_ID=101; Protocols = 4; Transports = TCP; Path = PATH; Pseudo = PSEUDO_PATH; Access_Type = RW; Attr_Expiration_Time = 0; Squash = no_root_squash; FSAL { Name = CEPH; User_Id = "ganesha"; Secret_Access_Key = CEPHXKEY; } } LOG { Default_Log_Level = "FULL_DEBUG"; } ------------ Does anyone have similar problems? Or if this behavior is by purpose, can you explain to me why this is the case? Thank you in advance for your time and thoughts. Kind regards, Michael

3 2

is ceph balancer doing anything?
by Andrei Mikhailovsky 28 Apr '20

28 Apr '20

Hello everyone, A few weeks ago I have enabled the ceph balancer on my cluster as per the instructions here: [ https://docs.ceph.com/docs/mimic/mgr/balancer/ | https://docs.ceph.com/docs/mimic/mgr/balancer/ ] I am running ceph version: ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) The cluster has 48 osds (40 osds in hdd pools and 8 osds in ssd pool) Currently, the balancer status is showing as Active. # ceph balancer status { "active": true, "plans": [], "mode": "upmap" } The health status of the cluster is: health: HEALTH_OK Previously, I've used the old REWEIGHT to change the placement of data as I've seen very uneven usage (ranging from about 60% usage on some OSDs to over 90% on others). So, I have a number of osds with reweight of 1 an some going down to 0.75. At the moment the osd usage ranges between about 65% to to just under 90%, so still a huge variation. After switching on the balancer, I've not actually seen any activity or data migration, so I am not sure if the balancer is working at all. Could someone tell me how do I check if balancing is doing its job? The second question is as the balancer is now switched on, do I suppose to set the reweight values back to their default value of 1? Many thanks

2 1

How to migrate ceph-xattribs?
by Frank Schilder 20 Apr '20

20 Apr '20

De all, we are in the process of migrating a ceph file system from a 2-pool layout (rep meta+ec data) to the recently recommended 3-pool layout (rep meta, per primary data, ec data). As part of this, we need to migrate any ceph xattribs set on files and directories. As these are no longer discoverable, how would one go about this? Special cases: How to migrate quota settings? How to migrate dir- and file-layouts? Ideally, at least quota attributes should be transferable on the fly with tools like rsync. If automatic migration is not possible, is there at least an efficient way to *find* everything with special ceph attributes? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 5

HELP! Ceph( v 14.2.8) bucket notification dose not work!
by 曹海旺 17 Apr '20

17 Apr '20

Hi, I upgrade the ceph from 14.2.7 to the new version 14.2.8 . The bucket notification dose not work. I can’t create a TOPIC : I use post man to send a post flow by https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic REQUEST: POST http://rgw1:7480/?Action=CreateTopic&Name=webno&push-endpoint=https://192.1… RESPONSE: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>MethodNotAllowed</Code> <RequestId>tx000000000000000000008-005e6a0eab-cbcad-bj</RequestId> <HostId>cbcad-bj-bjz</HostId> </Error> The debug info on the node below: 2020-03-12 18:49:24.684 7fdde1e1d700 1 ====== starting new request req=0x55c91a51e8f0 ===== 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s initializing for trans_id = tx00000000000000000000e-005e6a13b4-cbc6a-bj 2020-03-12 18:49:24.684 7fdde1e1d700 10 rgw api priority: s3=8 s3website=7 2020-03-12 18:49:24.684 7fdde1e1d700 10 host=192.168.3.250 2020-03-12 18:49:24.684 7fdde1e1d700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2020-03-12 18:49:24.684 7fdde1e1d700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ 2020-03-12 18:49:24.684 7fdde1e1d700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 2020-03-12 18:49:24.684 7fdde1e1d700 10 meta>> HTTP_X_AMZ_DATE 2020-03-12 18:49:24.684 7fdde1e1d700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2020-03-12 18:49:24.684 7fdde1e1d700 10 x>> x-amz-date:20200312T104924Z 2020-03-12 18:49:24.684 7fdde1e1d700 20 get_handler handler=26RGWHandler_REST_Service_S3 2020-03-12 18:49:24.684 7fdde1e1d700 10 handler=26RGWHandler_REST_Service_S3 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s getting op 4 2020-03-12 18:49:24.684 7fdde1e1d700 20 handler->ERRORHANDLER: err_no=-2003 new_err_no=-2003 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s http status=405 2020-03-12 18:49:24.684 7fdde1e1d700 1 ====== req done req=0x55c91a51e8f0 op status=0 http_status=405 latency=0s ====== 2020-03-12 18:49:25.502 7fde0fe79700 2 RGWDataChangesLog::ChangesRenewThread: start The same post works in the version 14.2.7 What is the correct way to create a Topic in version 14.2.8 ？

3 3

Re: MDS: obscene buffer_anon memory use when scanning lots of files
by John Madden 15 Apr '20

15 Apr '20

Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested: ~$ ceph tell mds.mds1 heap stats 2020-02-10 16:52:44.313 7fbda2cae700 0 client.59208005 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:44.337 7fbda3cb0700 0 client.59249562 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 tcmalloc heap stats:------------------------------------------------ MALLOC: 50000388656 (47684.1 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 174879528 ( 166.8 MiB) Bytes in central cache freelist MALLOC: + 14511680 ( 13.8 MiB) Bytes in transfer cache freelist MALLOC: + 14089320 ( 13.4 MiB) Bytes in thread cache freelists MALLOC: + 90534048 ( 86.3 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 50294403232 (47964.5 MiB) Actual memory used (physical + swap) MALLOC: + 50987008 ( 48.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 50345390240 (48013.1 MiB) Virtual address space used MALLOC: MALLOC: 260018 Spans in use MALLOC: 20 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ~$ ceph tell mds.mds1 heap release 2020-02-10 16:52:47.205 7f037eff5700 0 client.59249625 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:47.237 7f037fff7700 0 client.59249634 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 releasing free RAM back to system. The pools over 15 minutes or so: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2045, "bytes": 3069493686 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2445, "bytes": 3111162538 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 7850, "bytes": 7658678767 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 12274, "bytes": 11436728978 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 13747, "bytes": 11539478519 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 14615, "bytes": 13859676992 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 23267, "bytes": 22290063830 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 44944, "bytes": 40726959425 } And one about a minute after the heap release showing continued growth: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 50694, "bytes": 47343942094 } This is on a single active MDS with 2 standbys, scan for about a million files with about 20 parallel threads on two clients, open and read each if it exists. On Wed, Jan 22, 2020 at 8:25 AM John Madden <jmadden.com(a)gmail.com> wrote: > > > Couldn't John confirm that this is the issue by checking the heap stats and triggering the release via > > > > ceph tell mds.mds1 heap stats > > ceph tell mds.mds1 heap release > > > > (this would be much less disruptive than restarting the MDS) > > That was my first thought as well, but `release` doesn't appear to do > anything in this case. > > John

3 8

Fw: Incompatibilities (implicit_tenants & barbican) with Openstack after migrating from Ceph Luminous to Nautilus.
by Scheurer François 14 Apr '20

14 Apr '20

(resending to the new maillist) Dear Casey, Dear All, We tested the migration from Luminous to Nautilus and noticed two regressions breaking the RGW integration in Openstack: 1) the following config parameter is not working on Nautilus but is valid on Luminous and on Master: rgw_keystone_implicit_tenants = swift In the log: parse error setting 'rgw_keystone_implicit_tenants' to 'swift' (Expected option value to be integer, got 'swift') This param is important to make RGW working for S3 and Swift. Setting it to false breaks swift/openstack and setting it to true makes S3 incompatible with dns-style bucketnames (with shared or public access). Please note that path-style bucketnames are deprecated by AWS and most clients are only supporting dns-style... Ref.: https://tracker.ceph.com/issues/24348 https://github.com/ceph/ceph/commit/3ba7be8d1ac7ee43e69eebb58263cd080cca1d38 2) the server-side encryption (SSE-KMS) is broken on Nautilus: to reproduce the issue: s3cmd --access_key $ACCESSKEY --secret_key $SECRETKEY --host-bucket "%(bucket)s.$ENDPOINT" --host "$ENDPOINT" --region="$REGION" --signature-v2 --no-preserve --no-ssl --server-side-encryption --server-side-encryption-kms-id ${SECRET##*/} put helloenc.txt s3://testenc/ output: upload: 'helloenc.txt' -> 's3://testenc/helloenc.txt' [1 of 1] 9 of 9 100% in 0s 37.50 B/s done ERROR: S3 error: 403 (AccessDenied): Failed to retrieve the actual key, kms-keyid: cd0903db-c613-49be-96d9-165c02544bc7 rgw log: see below TLDR: after investigating, I found that radosgw was actually getting the barbican secret correctly but the HTTP CODE (=200) validation was failing because of a bug in Nautilus. My understanding is following (please correct me): The bug in src/rgw/rgw_http_client.cc . Since Nautilus HTTP_CODE are converted into ERROR_CODE (200 becomes 0) in the request processing. This happens in RGWHTTPManager::reqs_thread_entry(), which centralizes the processing of (curl) HTTP Requests with multi-treading. This is fine but the member variable http_status of the class RGWHTTPClient is not updated with the resulting HTTP CODE, so the variable keeps its initial value of 0. Then in src/rgw/rgw_crypt.cc the logic is still verifying that http_status is in range [200,299] and this fails... I wrote the following oneliner bugfix for src/rgw/rgw_http_client.cc: diff --git a/src/rgw/rgw_http_client.cc b/src/rgw/rgw_http_client.cc index d0f0baead6..7c115293ad 100644 --- a/src/rgw/rgw_http_client.cc +++ b/src/rgw/rgw_http_client.cc @@ -1146,6 +1146,7 @@ void *RGWHTTPManager::reqs_thread_entry() status = -EAGAIN; } int id = req_data->id; + req_data->client->http_status = http_status; finish_request(req_data, status); switch (result) { case CURLE_OK: The s3cmd is then working fine with KMS server side encryption. Questions: * Could someone please write a fix for the regression of 1) and make a PR ? * Could somebody also make a PR for 2? Thank you for your help. :-) Cheers Francois Scheurer rgw log: export CLUSTER=ceph; /home/local/ceph/build/bin/radosgw -f --cluster ${CLUSTER} --name client.rgw.$(hostname) --setuser ceph --setgroup ceph & tail -fn0 /var/log/ceph/ceph-client.rgw.ewos1-osd1-stage.log | less -IS 2020-02-26 16:32:59.208 7fc1f1c54700 20 Getting KMS encryption key for key=cd0903db-c613-49be-96d9-165c02544bc7 2020-02-26 16:32:59.208 7fc1f1c54700 20 Requesting secret from barbican url=http://keystone.service.stage.i.ewcs.ch:5000/v3/auth/tokens 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPClient::process: http_status: 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::process 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::send 2020-02-26 16:32:59.208 7fc1f1c54700 20 sending request to http://keystone.service.stage.i.ewcs.ch:5000/v3/auth/tokens 2020-02-26 16:32:59.208 7fc1f1c54700 20 ssl verification is set to off 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: client->init_request(req_data): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 register_request mgr=0x56374b865540 req_data->id=4, curl_handle=0x56374c77c4a0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::signal_thread(): write(thread_pipe[1], (void *)&buf, sizeof(buf)): 4 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: signal_thread(): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::send: rgw_http_manager->add_request(req): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::process: send(req): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : int wait() : ret: 0 2020-02-26 16:32:59.208 7fc2184a1700 20 link_request req_data=0x56374c96a240 req_data->id=4, curl_handle=0x56374c77c4a0 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: http_status: 201 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: rgw_http_error_to_errno(http_status): 0 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: finish_request(req_data, status): status: 0 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : void finish(int r) : ret: 0 2020-02-26 16:32:59.652 7fc1f1c54700 5 ewdebug: request_key_from_barbican: Accept application/octet-stream X-Auth-Token gAAAAABeVo-xxx 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPClient::process: http_status: 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::process 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::send 2020-02-26 16:32:59.652 7fc1f1c54700 20 sending request to http://barbican.service.stage.i.ewcs.ch:9311/v1/secrets/cd0903db-c613-49be-… 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: client->init_request(req_data): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 register_request mgr=0x56374b865540 req_data->id=5, curl_handle=0x56374c77c4a0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::signal_thread(): write(thread_pipe[1], (void *)&buf, sizeof(buf)): 4 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: signal_thread(): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::send: rgw_http_manager->add_request(req): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::process: send(req): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : int wait() : ret: 0 2020-02-26 16:32:59.652 7fc2184a1700 20 link_request req_data=0x56374c96a240 req_data->id=5, curl_handle=0x56374c77c4a0 => 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: http_status: 200 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: rgw_http_error_to_errno(http_status): 0 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: finish_request(req_data, status): status: 0 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : void finish(int r) : ret: 0 2020-02-26 16:32:59.752 7fc1f1c54700 5 ewdebug: request_key_from_barbican: secret_req.process: 0 => 2020-02-26 16:32:59.752 7fc1f1c54700 5 ewdebug: request_key_from_barbican: secret_req.get_http_status: 0 2020-02-26 16:32:59.752 7fc1f1c54700 5 ewdebug: request_key_from_barbican: secret_req.get_http_status not in [200,299] range! 2020-02-26 16:32:59.752 7fc1f1c54700 5 Failed to retrieve secret from barbican:cd0903db-c613-49be-96d9-165c02544bc7 2020-02-26 16:32:59.752 7fc1f1c54700 5 ERROR: failed to retrieve actual key from key_id: cd0903db-c613-49be-96d9-165c02544bc7 2020-02-26 16:32:59.752 7fc1f1c54700 2 req 1 1.092s s3:put_obj completing 2020-02-26 16:32:59.752 7fc1f1c54700 2 req 1 1.092s s3:put_obj op status=-13 2020-02-26 16:32:59.752 7fc1f1c54700 2 req 1 1.092s s3:put_obj http status=403 2020-02-26 16:32:59.752 7fc1f1c54700 1 ====== req done req=0x56374c9808d0 op status=-13 http_status=403 latency=1.092s ====== => we see that http_status is correct (200) but the variable secret_req.get_http_status (member of class RGWHTTPClient) is incorrect (0 instead of 200)

2 4

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

Load on drives of different sizes in ceph
by Andras Pataki 07 Apr '20

07 Apr '20

Hi cephers, I'm looking for some advice on what to do about drives of different sizes in the same cluster. We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this. I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? Andras

4 4

rgw multisite with https endpoints
by Richard Kearsley 06 Apr '20

06 Apr '20

Hi there I have a fairly simple ceph multisite configuration with 2 ceph clusters in 2 different datacenters in the same city The rgws have this config for ssl: rgw_frontends = civetweb port=7480+443s ssl_certificate=/opt/ssl/ceph-bundle.pem The certificate is a real issued certificate, not self signed I configured the multisite with the guide from https://docs.ceph.com/docs/nautilus/radosgw/multisite/ More or less ok so far, some learning curve but that's ok I can access and upload to buckets at both endpoints with s3 client using https - https://ceph01cs1.domain.com and https://ceph01cs2.domain.com - all good Now the problem seems to be when my zones in the zonegroup use https endpoints, e.g. { "id": "4c6774fb-01eb-41fe-a74a-c2693f8e69fc", "name": "eu", "api_name": "eu", "is_master": "true", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "zones": [ { "id": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "name": "ceph01cs1", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "fec1fec8-a3c1-454d-8ed2-2c1da45f9c33", "name": "ceph01cs2", "endpoints": [ "https://ceph01cs2.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "08921dd5-1523-41b6-908f-2f58aa38c969" } Meta syncs ok - buckets and users get created, but data doesn't, and period can be commited and appears on both clusters I can also curl between the two clusters over 443 However, data sync gets stuck on 'init': realm 08921dd5-1523-41b6-908f-2f58aa38c969 (world) zonegroup 4c6774fb-01eb-41fe-a74a-c2693f8e69fc (eu) zone 0c203df2-6f31-4ad1-a899-91f85bf34c4e (ceph01cs2) metadata sync no sync (zone is master) data sync source: fec1fec8-a3c1-454d-8ed2-2c1da45f9c33 (ceph01cs1) init full sync: 128/128 shards full sync: 0 buckets to sync incremental sync: 0/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] I find errors like: 2020-03-31 20:27:11.372 7f60c84e1700 0 RGW-SYNC:data:sync: ERROR: failed to init sync, retcode=-16 2020-03-31 20:27:29.548 7f60c84e1700 0 RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote data log shards 2020-03-31 20:29:48.499 7f60c94e3700 0 RGW-SYNC:meta: ERROR: failed to fetch all metadata keys If I change the endpoints in the zonegroup to plain http, e.g. http://ceph01cs1.domain.com:7480 and http://ceph01cs2.domain.com:7480 then sync starts! So my question, and I couldn't find any examples of people using https to sync.. are https endpoints supported with multisite? and why would meta work over https but not data? Many thanks Richard

3 3

v14.2.8 Nautilus released
by Abhishek Lekshmanan 06 Apr '20

06 Apr '20

This is the eighth update to the Ceph Nautilus release series. This release fixes issues across a range of subsystems. We recommend that all users upgrade to this release. Please note the following important changes in this release; as always the full changelog is posted at: https://ceph.io/releases/v14-2-8-nautilus-released Notable Changes --------------- * The default value of `bluestore_min_alloc_size_ssd` has been changed to 4K to improve performance across all workloads. * The following OSD memory config options related to bluestore cache autotuning can now be configured during runtime: - osd_memory_base (default: 768 MB) - osd_memory_cache_min (default: 128 MB) - osd_memory_expected_fragmentation (default: 0.15) - osd_memory_target (default: 4 GB) The above options can be set with:: ceph config set osd <option> <value> * The MGR now accepts `profile rbd` and `profile rbd-read-only` user caps. These caps can be used to provide users access to MGR-based RBD functionality such as `rbd perf image iostat` an `rbd perf image iotop`. * The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap balancing has been removed. Instead use the mgr balancer config `upmap_max_deviation` which now is an integer number of PGs of deviation from the target PGs per OSD. This can be set with a command like `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The default `upmap_max_deviation` is 1. There are situations where crush rules would not allow a pool to ever have completely balanced PGs. For example, if crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of the racks. In those cases, the configuration value can be increased. * RGW: a mismatch between the bucket notification documentation and the actual message format was fixed. This means that any endpoints receiving bucket notification, will now receive the same notifications inside a JSON array named 'Records'. Note that this does not affect pulling bucket notification from a subscription in a 'pubsub' zone, as these are already wrapped inside that array. * CephFS: multiple active MDS forward scrub is now rejected. Scrub currently only is permitted on a file system with a single rank. Reduce the ranks to one via `ceph fs set <fs_name> max_mds 1`. * Ceph now refuses to create a file system with a default EC data pool. For further explanation, see: https://docs.ceph.com/docs/nautilus/cephfs/createfs/#creating-pools * Ceph will now issue a health warning if a RADOS pool has a `pg_num` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.8.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 2d095e947a02261ce61424021bb43bd3022d35cb -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer HRB 21284 (AG Nürnberg)

6 7

Re: Questions on Ceph cluster without OS disks
by Martin Verges 05 Apr '20

05 Apr '20

Hello Samuel, we from croit.io don't use NFS to boot up Servers. We copy the OS directly into the RAM (approximately 0.5-1GB). Think of it like a container, you start it and throw it away when you no longer need it. This way we can save the slots of OS harddisks to add more storage per node and reduce overall costs as 1GB ram is cheaper then an OS disk and consumes less power. If our management node is down, nothing will happen to the cluster. No impact, no downtime. However, you do need the mgmt node to boot up the cluster. So after a very rare total power outage, your first system would be the mgmt node and then the cluster itself. But again, if you configure your systems correct, no manual work is required to recover from that. For everything else, it is possible (but definitely not needed) to deploy our mgmt node in active/passive HA. We have multiple hundred installations worldwide in production environments. Our strong PXE knowledge comes from more than 20 years of datacenter hosting experience and it never ever failed us in the last >10 years. The main benefits out of that: - Immutable OS freshly booted: Every host has exactly the same version, same library, kernel, Ceph versions,... - OS is heavily tested by us: Every croit deployment has exactly the same image. We can find errors much faster and hit much fewer errors. - Easy Update: Updating OS, Ceph or anything else is just a node reboot. No cluster downtime, No service Impact, full automatic handling by our mgmt Software. - No need to install OS: No maintenance costs, no labor required, no other OS management required. - Centralized Logs/Stats: As it is booted in memory, all logs and statistics are collected on a central place for easy access. - Easy to scale: It doesn't matter if you boot 3 oder 300 nodes, all boot the exact same image in a few seconds. .. lots more Please do not hesitate to contact us directly. We always try to offer an excellent service and are strongly customer oriented. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 21. März 2020 um 13:53 Uhr schrieb huxiaoyu(a)horebdata.cn < huxiaoyu(a)horebdata.cn>: > Hello， Martin， > > I notice that Croit advocate the use of ceph cluster without OS disks, but > with PXE boot. > > Do you use a NFS server to serve the root file system for each node? such > as hosting configuration files, user and password, log files, etc. My > question is, will the NFS server be a single point of failure? If the NFS > server goes down, the network experience any outage, ceph nodes may not be > able to write to the local file systems, possibly leading to service outage. > > How do you deal with the above potential issues in production? I am a bit > worried... > > best regards, > > samuel > > > > > ------------------------------ > huxiaoyu(a)horebdata.cn > > >

6 14

Is there a better way to make a samba/nfs gateway?
by Seth Galitzer 04 Apr '20

04 Apr '20

I have a hybrid environment and need to share with both Linux and Windows clients. For my previous iterations of file storage, I exported nfs and samba shares directly from my monolithic file server. All Linux clients used nfs and all Windows clients used samba. Now that I've switched to ceph, things are a bit more complicated. I built a gateway to export nfs and samba as needed, and connect that as a client to my ceph cluster. After having file locking problems with kernel nfs, I made the switch to nfs-ganesha, which has helped immensely. For Linux clients that have high I/O needs, like desktops and some web servers, I connect to ceph directly for those shares. For all other Linux needs, I use nfs from the gateway. For all Windows clients (desktops and a small number of servers), I use samba exported from the gateway. Since my ceph cluster went live in August, I have had some kind of strange (to me) error at least once a week, almost always related to the gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since we're on Spring Break at my university and not very busy, I decided to unmount/remount the ceph share, requiring stopping nfs and samba services. Stopping nfs-ganesha took a while, but it finally completed with no complaints from the ceph cluster. Stopping samba took longer and gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds. It finally finished, and I was able to unmount/remount the ceph share and that finally cleared all the errors. This is leading me to believe that samba on the gateway and all the clients attaching to that is putting a strain on the connection back to ceph. Which finally brings me to my question: is there a better way to export samba to my clients using the ceph back end? Or is this as good as it gets and I just have to put up with the seemingly frequent errors? I can live with the errors and have been able to handle them so far, but I know people who have much bigger clusters and many more clients than me (by an order of magnitude) and don't see nearly as many errors as I do. Which is why I'm trying to figure out what is special about my setup. All my ceph nodes are running latest nautilus on Centos 7 (I just updated last week to 14.2.8), as is the gateway host. I'm mounting ceph directly on the gateway (by way of the kernel using cephfs, not rados/rbd) to a single mount point and exporting from there. My searches so far have not turned up anything extraordinarily useful, so I'm asking for some guidance here. Any advice is welcome. Thanks. Seth -- Seth Galitzer Systems Coordinator Computer Science Department Kansas State University http://www.cs.ksu.edu/~sgsax sgsax(a)ksu.edu 785-532-7790

8 11

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

Odd CephFS Performance
by Gabryel Mason-Williams 03 Apr '20

03 Apr '20

We have been benchmarking CephFS and comparing it Rados to see the performance difference and how much overhead CephFS has. However, we are getting odd results when using more than 1 OSD server (each OSDS has only one disk) using CephFS but using Rados everything appears normal. These tests are run on the same Ceph Cluster. CephFS Rados OSDS Thread 16 Thread 16 1 289 316 2 139 546 3 143 728 4 142 844 CephFS is being benchmarked using: fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4M --numjobs=16 --size=1G --group_reporting Rados is being benchmarked using: rados bench -p cephfs_data 10 write -t 16 If you could provide some help or insight into why this is happening or how to stop it, that would be much appreciated. Kind regards, Gabryel

2 2

v15.2.0 Octopus released
by Abhishek Lekshmanan 03 Apr '20

03 Apr '20

We're happy to announce the first stable release of Octopus v15.2.0. There are a lot of changes and new features added, we advise everyone to read the release notes carefully, and in particular the upgrade notes, before upgrading. Please refer to the official blog entry https://ceph.io/releases/v15-2-0-octopus-released/ for a detailed version with links & changelog. This release wouldn't have been possible without the support of the community, this release saw contributions from over 330 developers & 80 organizations, and we thank everyone for making this release happen. Major Changes from Nautilus --------------------------- General ~~~~~~~ * A new deployment tool called **cephadm** has been introduced that integrates Ceph daemon deployment and management via containers into the orchestration layer. * Health alerts can now be muted, either temporarily or permanently. * Health alerts are now raised for recent Ceph daemons crashes. * A simple 'alerts' module has been introduced to send email health alerts for clusters deployed without the benefit of an existing external monitoring infrastructure. * Packages are built for the following distributions: - CentOS 8 - CentOS 7 (partial--see below) - Ubuntu 18.04 (Bionic) - Debian Buster - Container images (based on CentOS 8) Note that the dashboard, prometheus, and restful manager modules will not work on the CentOS 7 build due to Python 3 module dependencies that are missing in CentOS 7. Besides this packages built by the community will also available for the following distros: - Fedora (33/rawhide) - openSUSE (15.2, Tumbleweed) Dashboard ~~~~~~~~~ The mgr-dashboard has gained a lot of new features and functionality: * UI Enhancements - New vertical navigation bar - New unified sidebar: better background task and events notification - Shows all progress mgr module notifications - Multi-select on tables to perform bulk operations * Dashboard user account security enhancements - Disabling/enabling existing user accounts - Clone an existing user role - Users can change their own password - Configurable password policies: Minimum password complexity/length requirements - Configurable password expiration - Change password after first login New and enhanced management of Ceph features/services: * OSD/device management - List all disks associated with an OSD - Add support for blinking enclosure LEDs via the orchestrator - List all hosts known by the orchestrator - List all disks and their properties attached to a node - Display disk health information (health prediction and SMART data) - Deploy new OSDs on new disks/hosts - Display and allow sorting by an OSD's default device class in the OSD table - Explicitly set/change the device class of an OSD, display and sort OSDs by device class * Pool management - Viewing and setting pool quotas - Define and change per-pool PG autoscaling mode * RGW management enhancements - Enable bucket versioning - Enable MFA support - Select placement target on bucket creation * CephFS management enhancements - CephFS client eviction - CephFS snapshot management - CephFS quota management - Browse CephFS directory * iSCSI management enhancements - Show iSCSI GW status on landing page - Prevent deletion of IQNs with open sessions - Display iSCSI "logged in" info * Prometheus alert management - List configured Prometheus alerts RADOS ~~~~~ * Objects can now be brought in sync during recovery by copying only the modified portion of the object, reducing tail latencies during recovery. * Ceph will allow recovery below *min_size* for Erasure coded pools, wherever possible. * The PG autoscaler feature introduced in Nautilus is enabled for new pools by default, allowing new clusters to autotune *pg num* without any user intervention. The default values for new pools and RGW/CephFS metadata pools have also been adjusted to perform well for most users. * BlueStore has received several improvements and performance updates, including improved accounting for "omap" (key/value) object data by pool, improved cache memory management, and a reduced allocation unit size for SSD devices. (Note that by default, the first time each OSD starts after upgrading to octopus it will trigger a conversion that may take from a few minutes to a few hours, depending on the amount of stored "omap" data.) * Snapshot trimming metadata is now managed in a more efficient and scalable fashion. RBD block storage ~~~~~~~~~~~~~~~~~ * Mirroring now supports a new snapshot-based mode that no longer requires the journaling feature and its related impacts in exchange for the loss of point-in-time consistency (it remains crash consistent). * Clone operations now preserve the sparseness of the underlying RBD image. * The trash feature has been improved to (optionally) automatically move old parent images to the trash when their children are all deleted or flattened. * The trash can be configured to automatically purge on a defined schedule. * Images can be online re-sparsified to reduce the usage of zeroed extents. * The `rbd-nbd` tool has been improved to use more modern kernel interfaces. * Caching has been improved to be more efficient and performant. * `rbd-mirror` automatically adjusts its per-image memory usage based upon its memory target. * A new persistent read-only caching daemon is available to offload reads from shared parent images. RGW object storage ~~~~~~~~~~~~~~~~~~ * New `Multisite Sync Policy` primitives for per-bucket replication. (EXPERIMENTAL) * S3 feature support: - Bucket Replication (EXPERIMENTAL) - `Bucket Notifications`_ via HTTP/S, AMQP and Kafka - Bucket Tagging - Object Lock - Public Access Block for buckets * Bucket sharding: - Significantly improved listing performance on buckets with many shards. - Dynamic resharding prefers prime shard counts for improved distribution. - Raised the default number of bucket shards to 11. * Added `HashiCorp Vault Integration`_ for SSE-KMS. * Added Keystone token cache for S3 requests. CephFS distributed file system ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Inline data support in CephFS has been deprecated and will likely be removed in a future release. * MDS daemons can now be assigned to manage a particular file system via the new `mds_join_fs` option. * MDS now aggressively asks idle clients to trim caps which improves stability when file system load changes. * The mgr volumes plugin has received numerous improvements to support CephFS via CSI, including snapshots and cloning. * cephfs-shell has had numerous incremental improvements and bug fixes. Upgrading from Mimic or Nautilus -------------------------------- You can monitor the progress of your upgrade at each stage with the `ceph versions` command, which will tell you what ceph version(s) are running for each type of daemon. Instructions ~~~~~~~~~~~~ #. Make sure your cluster is stable and healthy (no down or recovering OSDs). (Optional, but recommended.) #. Set the `noout` flag for the duration of the upgrade. (Optional, but recommended.):: # ceph osd set noout #. Upgrade monitors by installing the new packages and restarting the monitor daemons. For example, on each monitor host,:: # systemctl restart ceph-mon.target Once all monitors are up, verify that the monitor upgrade is complete by looking for the `octopus` string in the mon map. The command:: # ceph mon dump | grep min_mon_release should report:: min_mon_release 15 (nautilus) If it doesn't, that implies that one or more monitors hasn't been upgraded and restarted and/or the quorum does not include all monitors. #. Upgrade `ceph-mgr` daemons by installing the new packages and restarting all manager daemons. For example, on each manager host,:: # systemctl restart ceph-mgr.target Verify the `ceph-mgr` daemons are running by checking `ceph -s`:: # ceph -s ... services: mon: 3 daemons, quorum foo,bar,baz mgr: foo(active), standbys: bar, baz ... #. Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts:: # systemctl restart ceph-osd.target Note that the first time each OSD starts, it will do a format conversion to improve the accounting for "omap" data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data). You can disable this automatic conversion with:: # ceph config set osd bluestore_fsck_quick_fix_on_mount false You can monitor the progress of the OSD upgrades with the `ceph versions` or `ceph osd versions` commands:: # ceph osd versions { "ceph version 13.2.5 (...) mimic (stable)": 12, "ceph version 15.2.0 (...) octopus (stable)": 22, } #. Upgrade all CephFS MDS daemons. For each CephFS file system, #. Reduce the number of ranks to 1. (Make note of the original number of MDS daemons first if you plan to restore it later.):: # ceph status # ceph fs set <fs_name> max_mds 1 #. Wait for the cluster to deactivate any non-zero ranks by periodically checking the status:: # ceph status #. Take all standby MDS daemons offline on the appropriate hosts with:: # systemctl stop ceph-mds@<daemon_name> #. Confirm that only one MDS is online and is rank 0 for your FS:: # ceph status #. Upgrade the last remaining MDS daemon by installing the new packages and restarting the daemon:: # systemctl restart ceph-mds.target #. Restart all standby MDS daemons that were taken offline:: # systemctl start ceph-mds.target #. Restore the original value of `max_mds` for the volume:: # ceph fs set <fs_name> max_mds <original_max_mds> #. Upgrade all radosgw daemons by upgrading packages and restarting daemons on all hosts:: # systemctl restart ceph-radosgw.target #. Complete the upgrade by disallowing pre-Octopus OSDs and enabling all new Octopus-only functionality:: # ceph osd require-osd-release octopus #. If you set `noout` at the beginning, be sure to clear it with:: # ceph osd unset noout #. Verify the cluster is healthy with `ceph health`. If your CRUSH tunables are older than Hammer, Ceph will now issue a health warning. If you see a health alert to that effect, you can revert this change with:: ceph config set mon mon_crush_min_required_version firefly If Ceph does not complain, however, then we recommend you also switch any existing CRUSH buckets to straw2, which was added back in the Hammer release. If you have any 'straw' buckets, this will result in a modest amount of data movement, but generally nothing too severe.:: ceph osd getcrushmap -o backup-crushmap ceph osd crush set-all-straw-buckets-to-straw2 If there are problems, you can easily revert with:: ceph osd setcrushmap -i backup-crushmap Moving to 'straw2' buckets will unlock a few recent features, like the `crush-compat` :ref:`balancer <balancer>` mode added back in Luminous. #. If you are upgrading from Mimic, or did not already do so when you upgraded to Nautlius, we recommened you enable the new :ref:`v2 network protocol <msgr2>`, issue the following command:: ceph mon enable-msgr2 This will instruct all monitors that bind to the old default port 6789 for the legacy v1 protocol to also bind to the new 3300 v2 protocol port. To see if all monitors have been updated,:: ceph mon dump and verify that each monitor has both a `v2:` and `v1:` address listed. #. Consider enabling the :ref:`telemetry module <telemetry>` to send anonymized usage statistics and crash information to the Ceph upstream developers. To see what would be reported (without actually sending any information to anyone),:: ceph mgr module enable telemetry ceph telemetry show If you are comfortable with the data that is reported, you can opt-in to automatically report the high-level cluster metadata with:: ceph telemetry on For more information about the telemetry module, see :ref:`the documentation <telemetry>`. Upgrading from pre-Mimic releases (like Luminous) ------------------------------------------------- You *must* first upgrade to Mimic (13.2.z) or Nautilus (14.2.z) before upgrading to Octopus. Upgrade compatibility notes --------------------------- * Starting with Octopus, there is now a separate repository directory for each version on `download.ceph.com` (e.g., `rpm-15.2.0` and `debian-15.2.0`). The traditional package directory that is named after the release (e.g., `rpm-octopus` and `debian-octopus`) is now a symlink to the most recently bug fix version for that release. We no longer generate a single repository that combines all bug fix versions for a single named release. * The RGW "num_rados_handles" has been removed. If you were using a value of "num_rados_handles" greater than 1 multiply your current "objecter_inflight_ops" and "objecter_inflight_op_bytes" paramaeters by the old "num_rados_handles" to get the same throttle behavior. * Ceph now packages python bindings for python3.6 instead of python3.4, because python3 in EL7/EL8 is now using python3.6 as the native python3. see the `announcement`_ for more details on the background of this change. * librbd now uses a write-around cache policy be default, replacing the previous write-back cache policy default. This cache policy allows librbd to immediately complete write IOs while they are still in-flight to the OSDs. Subsequent flush requests will ensure all in-flight write IOs are completed prior to completing. The librbd cache policy can be controlled via a new "rbd_cache_policy" configuration option. * librbd now includes a simple IO scheduler which attempts to batch together multiple IOs against the same backing RBD data block object. The librbd IO scheduler policy can be controlled via a new "rbd_io_scheduler" configuration option. * RGW: radosgw-admin introduces two subcommands that allow the managing of expire-stale objects that might be left behind after a bucket reshard in earlier versions of RGW. One subcommand lists such objects and the other deletes them. Read the troubleshooting section of the dynamic resharding docs for details. * RGW: Bucket naming restrictions have changed and likely to cause InvalidBucketName errors. We recommend to set `rgw_relaxed_s3_bucket_names` option to true as a workaround. * In the Zabbix Mgr Module there was a typo in the key being send to Zabbix for PGs in backfill_wait state. The key that was sent was 'wait_backfill' and the correct name is 'backfill_wait'. Update your Zabbix template accordingly so that it accepts the new key being send to Zabbix. * zabbix plugin for ceph manager now includes osd and pool discovery. Update of zabbix_template.xml is needed to receive per-pool (read/write throughput, diskspace usage) and per-osd (latency, status, pgs) statistics * The format of all date + time stamps has been modified to fully conform to ISO 8601. The old format (`YYYY-MM-DD HH:MM:SS.ssssss`) excluded the `T` separator between the date and time and was rendered using the local time zone without any explicit indication. The new format includes the separator as well as a `+nnnn` or `-nnnn` suffix to indicate the time zone, or a `Z` suffix if the time is UTC. For example, `2019-04-26T18:40:06.225953+0100`. Any code or scripts that was previously parsing date and/or time values from the JSON or XML structure CLI output should be checked to ensure it can handle ISO 8601 conformant values. Any code parsing date or time values from the unstructured human-readable output should be modified to parse the structured output instead, as the human-readable output may change without notice. * The `bluestore_no_per_pool_stats_tolerance` config option has been replaced with `bluestore_fsck_error_on_no_per_pool_stats` (default: false). The overall default behavior has not changed: fsck will warn but not fail on legacy stores, and repair will convert to per-pool stats. * The disaster-recovery related 'ceph mon sync force' command has been replaced with 'ceph daemon <...> sync_force'. * The `osd_recovery_max_active` option now has `osd_recovery_max_active_hdd` and `osd_recovery_max_active_ssd` variants, each with different default values for HDD and SSD-backed OSDs, respectively. By default `osd_recovery_max_active` now defaults to zero, which means that the OSD will conditionally use the HDD or SSD option values. Administrators who have customized this value may want to consider whether they have set this to a value similar to the new defaults (3 for HDDs and 10 for SSDs) and, if so, remove the option from their configuration entirely. * monitors now have a `ceph osd info` command that will provide information on all osds, or provided osds, thus simplifying the process of having to parse `osd dump` for the same information. * The structured output of `ceph status` or `ceph -s` is now more concise, particularly the `mgrmap` and `monmap` sections, and the structure of the `osdmap` section has been cleaned up. * A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. New admin command `ceph daemon mgr.# dump_osd_network [threshold]` command will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. New admin command `ceph daemon osd.# dump_osd_network [threshold]` will do the same but only including heartbeats initiated by the specified OSD. * Inline data support for CephFS has been deprecated. When setting the flag, users will see a warning to that effect, and enabling it now requires the `--yes-i-really-really-mean-it` flag. If the MDS is started on a filesystem that has it enabled, a health warning is generated. Support for this feature will be removed in a future release. * `ceph {set,unset} full` is not supported anymore. We have been using `full` and `nearfull` flags in OSD map for tracking the fullness status of a cluster back since the Hammer release, if the OSD map is marked `full` all write operations will be blocked until this flag is removed. In the Infernalis release and Linux kernel 4.7 client, we introduced the per-pool full/nearfull flags to track the status for a finer-grained control, so the clients will hold the write operations if either the cluster-wide `full` flag or the per-pool `full` flag is set. This was a compromise, as we needed to support the cluster with and without per-pool `full` flags support. But this practically defeated the purpose of introducing the per-pool flags. So, in the Mimic release, the new flags finally took the place of their cluster-wide counterparts, as the monitor started removing these two flags from OSD map. So the clients of Infernalis and up can benefit from this change, as they won't be blocked by the full pools which they are not writing to. In this release, `ceph {set,unset} full` is now considered as an invalid command. And the clients will continue honoring both the cluster-wide and per-pool flags to be backward comaptible with pre-infernalis clusters. * The telemetry module now reports more information. First, there is a new 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in:: ceph config set mgr mgr/telemetry/channel_device false Second, we now report more information about CephFS file systems, including: - how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached We have also added: - which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use If you had telemetry enabled, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: $ ceph telemetry show # see everything $ ceph telemetry show basic # basic cluster info (including all of the new info) * Following invalid settings now are not tolerated anymore for the command `ceph osd erasure-code-profile set xxx`. * invalid `m` for "reed_sol_r6_op" erasure technique * invalid `m` and invalid `w` for "liber8tion" erasure technique * New OSD daemon command dump_recovery_reservations which reveals the recovery locks held (in_progress) and waiting in priority queues. * New OSD daemon command dump_scrub_reservations which reveals the scrub reservations that are held for local (primary) and remote (replica) PGs. * Previously, `ceph tell mgr ...` could be used to call commands implemented by mgr modules. This is no longer supported. Since luminous, using `tell` has not been necessary: those same commands are also accessible without the `tell mgr` portion (e.g., `ceph tell mgr influx foo` is the same as `ceph influx foo`. `ceph tell mgr ...` will now call admin commands--the same set of commands accessible via `ceph daemon ...` when you are logged into the appropriate host. * The `ceph tell` and `ceph daemon` commands have been unified, such that all such commands are accessible via either interface. Note that ceph-mgr tell commands are accessible via either `ceph tell mgr ...` or `ceph tell mgr.<id> ...`, and it is only possible to send tell commands to the active daemon (the standbys do not accept incoming connections over the network). * Ceph will now issue a health warning if a RADOS pool as a `pg_num` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false * The format of MDSs in `ceph fs dump` has changed. * The `mds_cache_size` config option is completely removed. Since luminous, the `mds_cache_memory_limit` config option has been preferred to configure the MDS's cache limits. * The `pg_autoscale_mode` is now set to `on` by default for newly created pools, which means that Ceph will automatically manage the number of PGs. To change this behavior, or to learn more about PG autoscaling, see :ref:`pg-autoscaler`. Note that existing pools in upgraded clusters will still be set to `warn` by default. * The `upmap_max_iterations` config option of mgr/balancer has been renamed to `upmap_max_optimizations` to better match its behaviour. * `mClockClientQueue` and `mClockClassQueue` OpQueue implementations have been removed in favor of of a single `mClockScheduler` implementation of a simpler OSD interface. Accordingly, the `osd_op_queue_mclock*` family of config options has been removed in favor of the `osd_mclock_scheduler*` family of options. * The config subsystem now searches dot ('.') delimited prefixes for options. That means for an entity like `client.foo.bar`, its overall configuration will be a combination of the global options, `client`, `client.foo`, and `client.foo.bar`. Previously, only global, `client`, and `client.foo.bar` options would apply. This change may affect the configuration for clients that include a `.` in their name. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.0.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer

14 21

Logging remove duplicate time
by Marc Roos 02 Apr '20

02 Apr '20

I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean; I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean; I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean; I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean;

2 2

Re: Netplan bonding configuration
by James McEwan 02 Apr '20

02 Apr '20

Hi, I am currently building a 10 node Ceph cluster, each OSD node has 2x 25 Gbit/s nics, and I have 2 TOR switches (mlag not supported). enp179s0f0 -> sw1 enp179s0f1 -> sw2 vlan 323 is used for ‘public network’ vlan 324 is used for ‘cluster network’ My desired configuration is to create two bond interfaces in active-backup mode: bond0 - enp179s0f0.323 (active) - enp179s0f1.323 (backup) bond1 - enp179s0f0.324 (backup) - enp179s0f1.324 (active) This way, the public network will use switch1, and the cluster network will use switch2, under normal operation. I am, however, having an issue implementing this configuration in Ubuntu 18.04 with netplan (see configuration at the end of this post). When I reboot a node with the below netplan configuration, the bond interface is created, but the vlan interfaces are not added to the bond. I see the following errors in the log: systemd-networkd[1641]: enp179s0f0.323: Enslaving by 'bond0’ systemd-networkd[1641]: bond0: Enslaving link 'enp179s0f0.323’ systemd-networkd[1641]: enp179s0f1.323: Enslaving by 'bond0’ systemd-networkd[1641]: bond0: Enslaving link 'enp179s0f1.323’ systemd-networkd[1643]: enp179s0f1.323: Could not join netdev: Operation not permitted systemd-networkd[1643]: enp179s0f1.323: Failed systemd-networkd[1643]: enp179s0f0.323: Could not join netdev: Operation not permitted systemd-networkd[1643]: enp179s0f0.323: Failed If I manually run ’systemctl restart systemd-networkd’ after boot has completed, then the bond is successfully created with the vlan interfaces. Does anybody have a similar configuration working specifically with netplan/networkd? Could you please share your configuration? Netplan config that doesn’t work at boot time: network: version: 2 renderer: networkd ethernets: enp179s0f0: {} enp179s0f1: {} bonds: bond0: dhcp4: false dhcp6: false interfaces: - enp179s0f0.323 - enp179s0f1.323 parameters: mode: active-backup primary: enp179s0f0.323 mii-monitor-interval: 1 addresses: [insert address here] bond1: dhcp4: false dhcp6: false interfaces: - enp179s0f0.324 - enp179s0f1.324 parameters: mode: active-backup primary: enp179s0f1.324 mii-monitor-interval: 1 addresses: [insert address here] vlans: enp179s0f0.323: id: 323 link: enp179s0f0 enp179s0f1.323: id: 323 link: enp179s0f1 enp179s0f0.324: id: 324 link: enp179s0f0 enp179s0f1.324: id: 324 link: enp179s0f1

5 7

Replace OSD node without remapping PGs
by Nghia Viet Tran 02 Apr '20

02 Apr '20

Hi everyone, I'm working on replacing OSDs node with the newer one. The new host has the new hostname and new disk (faster one but the same size with old disk). My plan is - Reweight the OSD to zero to spread all existed data to the rest nodes to keep data availability - set flag noout, norebalance, norecover, nobackfill, destroy the OSD and join the new OSD as the same ID of the old one. By above approach, the cluster will remap PGs of all nodes. Each data will be moved twice times until it reach the new OSD (reweight and join new node as same ID) I also did the other way that only set flags and destroy OSD. But the result is still the same (degraded objects from destroyed osd and misplaced object after joining new osd) Are there any ways to replace the OSD node directly without remapping PGs of the whole cluster? Many thanks! Nghia.

3 3

Using sendfile on Ceph FS results in data stuck in client cache
by Mikael Öhman 01 Apr '20

01 Apr '20

Hi all, Using sendfile function to write data to cephfs, the data doesn't end up being written. From the client that writes the file, it looks correct at first, but from all other ceph clients, the size is 0 bytes. Re-mounting the filesystem, the data is lost. I didn't see any errors, the data just doesn't get written, as if it's just cached in cephfs client. Writing just an extra byte at the end of the file (without sendfile), it seems to trigger the actual write of all the data. Could someone else confirm if they are also seeing such issue? I'm on ceph 13.2.8, using kernel module for mounting on CentOS7. I've used this sendfile-example for the example below: https://github.com/pijewski/sendfile-example/blob/master/sendfile.c Using a small 27 byte source file. # ls -lh examples/ -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src # ./sendfile examples/src examples/dst 27 # ls -lh examples/ ------x--- 1 root c3-staff 27 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src But, directory is still on 27 bytes: # ls -lhd examples drwxr-sr-x 1 root c3-staff 27 Mar 24 18:15 examples and on all other cephfs clients, the file is empty: # ls -lh examples/ ------x--- 1 root c3-staff 0 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src Is this a bug in cephfs, or should I not expect sendfile to work (as it is not posix compliant). There are no error reported from what i can see, and it is 100% reproducible Best regards, Mikael

2 4

osd can not start at boot after upgrade to octopus
by Lomayani S. Laizer 01 Apr '20

01 Apr '20

Hello, I have upgraded nautilus cluster to octopus few days ago. the cluster was running ok and even after to octopus everything was running ok the issue came when i rebooted the servers for updating the kernel. Two servers out of 6 osd's servers osd cant start. No error reported in ceph-volume.log and ceph-volume-systemd.log Starting osd with /usr/bin/ceph-osd -f --cluster ceph --id 30 --setuser ceph --setgroup ceph works just fine. the issue is starting osd in systemd ceph-volume-systemd.log 16:36:28,193][systemd][WARNING] failed activating OSD, retries left: 30 [2020-03-31 16:36:28,196][systemd][WARNING] command returned non-zero exit status: 1 [2020-03-31 16:36:28,196][systemd][WARNING] failed activating OSD, retries left: 30 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-31-a8efb7db-686b-4789-a9c4-01442c28577f [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 31-a8efb7db-686b-4789-a9c4-01442c28577f [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: lvm-29-3e52d340-5416-46e6-b697-c15ca85f6883 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: lvm-32-3841a62d-d6bc-404a-8762-163530b2d5d4 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 29-3e52d340-5416-46e6-b697-c15ca85f6883 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 32-3841a62d-d6bc-404a-8762-163530b2d5d4 [2020-03-31 16:41:25,108][ceph_volume.process][INFO ] Running command: /usr/sbin/ceph-volume lvm trigger 29-3e52d340-5416-46e6-b697-c15ca85f6883 ceph-volume.log 2-163530b2d5d4 [2020-03-31 17:17:23,679][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@31 [2020-03-31 17:17:23,863][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@30 [2020-03-31 17:17:24,045][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@33 [2020-03-31 17:17:24,241][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@32 [2020-03-31 17:17:24,449][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@28 [2020-03-31 17:17:24,629][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@29 [2020-03-31 17:17:24,652][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)31.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:24,664][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)30.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:24,872][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@31 [2020-03-31 17:17:24,875][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)33.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,072][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)32.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,075][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@30 [2020-03-31 17:17:25,282][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@33 [2020-03-31 17:17:25,497][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)28.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,499][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@32 [2020-03-31 17:17:25,520][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)29.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,705][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@28 [2020-03-31 17:17:25,887][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@29 -- Lomayani

2 2

No reply or very slow reply from Prometheus plugin - ceph-mgr 13.2.8 mimic
by Paul Choi 01 Apr '20

01 Apr '20

Hello, We are running Mimic 13.2.8 with our cluster, and since upgrading to 13.2.8 the Prometheus plugin seems to hang a lot. It used to respond under 10s but now it often hangs. Restarting the mgr processes helps temporarily but within minutes it gets stuck again. The active mgr doesn't exit when doing `systemctl stop ceph-mgr.target" and needs to be kill -9'ed. Is there anything I can do to address this issue, or at least get better visibility into the issue? We only have a few plugins enabled: $ ceph mgr module ls { "enabled_modules": [ "balancer", "prometheus", "zabbix" ], 3 mgr processes, but it's a pretty large cluster (near 4000 OSDs) and it's a busy one with lots of rebalancing. (I don't know if a busy cluster would seriously affect the mgr's performance, but just throwing it out there) services: mon: 5 daemons, quorum woodenbox0,woodenbox2,woodenbox4,woodenbox3,woodenbox1 mgr: woodenbox2(active), standbys: woodenbox0, woodenbox1 mds: cephfs-1/1/1 up {0=woodenbox6=up:active}, 1 up:standby-replay osd: 3964 osds: 3928 up, 3928 in; 831 remapped pgs rgw: 4 daemons active Thanks in advance for your help, -Paul Choi

4 13

Multiple CephFS creation
by Jarett DeAngelis 01 Apr '20

01 Apr '20

Hi guys, This is documented as an experimental feature, but it doesn’t explain how to ensure that affinity for a given MDS sticks to the second filesystem you create. Has anyone had success implementing a second CephFS? In my case it will be based on a completely different pool from my first one. Thanks. J

4 8

Re: Questions on Ceph cluster without OS disks
by Eric Petit 31 Mar '20

31 Mar '20

It works well for me, been running a couple clusters for 1-2 years where all OSD hosts (~200) have no system disks and instead netboot from PXE. No NFS server involved, each host loads the same system image (Debian Live squashfs) into memory on boot and runs independently from there on out. Takes some trickery to configure and bring the OSDs up on boot (using puppet in my case), though that might get easier with the containerized approach in Ceph 15+. Best, Eric > On 21 Mar 2020, at 14:18, huxiaoyu(a)horebdata.cn wrote: > > Hi, Marc, > > Indeed PXE boot makes a lot sense in large cluster, cuting down OS deployment and management burden, but only iff no single of failure is guaranteed... > > best regards, > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: Marc Roos > Date: 2020-03-21 14:13 > To: ceph-users; huxiaoyu; martin.verges > Subject: RE: [ceph-users] Questions on Ceph cluster without OS disks > > I would say it is not a 'proven technology' otherwise you would see a > wide spread implementation and adaptation of this method. However if you > really need the physical disk space, it is a solution. Although I also > would have questions on creating an extra redundant environment to > service remote booting, just to spare a os disk position. Maybe this > makes more sence in really big environments. > > > > > > -----Original Message----- > From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] > Sent: 21 March 2020 13:54 > To: Martin Verges; ceph-users > Subject: [ceph-users] Questions on Ceph cluster without OS disks > > Hello， Martin， > > I notice that Croit advocate the use of ceph cluster without OS disks, > but with PXE boot. > > Do you use a NFS server to serve the root file system for each node? > such as hosting configuration files, user and password, log files, etc. > My question is, will the NFS server be a single point of failure? If the > NFS server goes down, the network experience any outage, ceph nodes may > not be able to write to the local file systems, possibly leading to > service outage. > > How do you deal with the above potential issues in production? I am a > bit worried... > > best regards, > > samuel > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph WAL/DB disks - do they really only use 3GB, or 30Gb, or 300GB
by victorhooi＠yahoo.com 31 Mar '20

31 Mar '20

Hi, I'm using Intel Optane disks to provide WAL/DB capacity for my Ceph cluster (which is part of Proxmox - for VM hosting). I've read that WAL/DB partitions only use either 3GB, or 30GB, or 300GB - due to the way that RocksDB works. Is this true? My current partition for WAL/DB is 145 GB - does this mean that 115Gb of that will be permanently wasted? Is this behaviour documented somewhere, or is there some background, so I can understand a bit more about how it works? Thanks, Victor

2 1

Stop logging ceph-mgr every 2s
by Marc Roos 31 Mar '20

31 Mar '20

How to get rid of this logging?? Mar 31 13:40:03 c01 ceph-mgr: 2020-03-31 13:40:03.521 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672067: 384 pgs: 384 active+clean;

1 0

[ceph][radosgw] nautilus multisite problems
by Ignazio Cassano 31 Mar '20

31 Mar '20

Hello, I have configured a multisite ceph. The master zone has not changed but on the destination zone I had some problems. On the destination zone I cleaned and reinstalled the radosgw, but trying to assing the same zone name it had before reinstallation does not work (radosgw does not start). I changed its zone name and radosgw starts, but the master is using the old name when I try to execute: radosgw-admin period update commit Please, how can I solve this problem ? Thanks Ignazi

2 2

samba ceph-vfs and scrubbing interval
by Marco Savoca 31 Mar '20

31 Mar '20

Hi all, i‘m running a 3 node ceph cluster setup with collocated mons and mds for actually 3 filesystems at home since mimic. I’m planning to downgrade to one FS and use RBD in the future, but this is another story. I’m using the cluster as cold storage on spindles with EC-pools for archive purposes. The cluster usually does not run 24/7. I actually managed to upgrade to octopus without problems yesterday. So first of all: great job with the release. Now I have a little problem and a general question to address. I have tried to share the CephFS via samba and the ceph-vfs module but I could not manage to get write access (read access is not a problem) to the share (even with the admin key). When I share the mounted path (kernel module or fuser mount) instead as usual there are no problems at all. Is ceph-vfs generally read only and I missed this point? Furthermore I suppose, that there is no possibility to choose between the different mds namespaces, right? Now the general question. Since the cluster does not run 24/7 as stated and is turned on perhaps once a week for a couple of hours on demand, what are reasonable settings for the scrubbing intervals? As I said, the storage is cold and there is mostly read i/o. The archiving process adds approximately 0.5 % of new data of the cluster’s total storage capacity. Stay healthy and regards, Marco Savoca

3 3

Unable to use iscsi gateway with https | iscsi-gateway-add returns errors
by givemeone 31 Mar '20

31 Mar '20

Hi all, I am installing ceph Nautilus and getting constantly errors while adding iscsi gateways It was working using http schema but after moving to https with wildcard certs gives API errors Below some of my configurations Thanks for your help Command: ceph --cluster ceph dashboard iscsi-gateway-add https://myadmin:admin.01@1.2.3.4:5050 Error: Error EINVAL: iscsi REST API cannot be reached. Please check your configuration and that the API endpoint is accessible Tried also disabling ssl verify # ceph dashboard set-rgw-api-ssl-verify False Option RGW_API_SSL_VERIFY updated "/etc/ceph/iscsi-gateway.cfg" 23L, 977C # Ansible managed [config] api_password = admin.01 api_port = 5050 # API settings. # The API supports a number of options that allow you to tailor it to your # local environment. If you want to run the API under https, you will need to # create cert/key files that are compatible for each iSCSI gateway node, that is # not locked to a specific node. SSL cert and key files *must* be called # 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory # on *each* gateway node. With the SSL files in place, you can use 'api_secure = true' # to switch to https mode. # To support the API, the bear minimum settings are: api_secure = True # Optional settings related to the CLI/API service api_user = myadmin cluster_name = ceph loop_delay = 1 trusted_ip_list = 1.2.3.3,1.2.3.4 Log file ====== ceph-rgw-cnode04.rgw0.log 2020-03-30 10:24:20.392 7f6a2dc1b700 1 ====== req done req=0x561d9ce465f0 op status=0 http_status=200 latency=0.0119993s ====== 2020-03-30 10:24:20.394 7f6a2cc19700 1 ====== starting new request req=0x561d9ce465f0 ===== 2020-03-30 10:24:20.396 7f6a2cc19700 1 ====== req done req=0x561d9ce465f0 op status=0 http_status=404 latency=0.00199988s ====== 2020-03-30 10:24:20.397 7f6a2bc17700 1 ====== starting new request req=0x561d9ce465f0 ===== 2020-03-30 10:24:20.410 7f6a2bc17700 1 ====== req done req=0x561d9ce465f0 op status=0 http_status=200 latency=0.0129992s ====== 2020-03-30 10:24:20.499 7f6a27c0f700 1 ====== starting new request req=0x561d9cec25f0 ===== 2020-03-30 10:24:20.502 7f6a27c0f700 1 ====== req done req=0x561d9cec25f0 op status=0 http_status=200 latency=0.00299982s ====== 2020-03-30 10:24:20.504 7f6a2740e700 1 ====== starting new request req=0x561d9cec25f0 ===== 2020-03-30 10:24:20.506 7f6a2740e700 1 ====== req done req=0x561d9cec25f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.516 7f6a22404700 1 ====== starting new request req=0x561d9cf825f0 ===== 2020-03-30 10:24:30.518 7f6a22404700 1 ====== req done req=0x561d9cf825f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.620 7f6a1ebfd700 1 ====== starting new request req=0x561d9cf925f0 ===== 2020-03-30 10:24:30.622 7f6a1ebfd700 1 ====== req done req=0x561d9cf925f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.708 7f6a19bf3700 1 ====== starting new request req=0x561d9cfd45f0 ===== 2020-03-30 10:24:30.708 7f6a193f2700 1 ====== starting new request req=0x561d9cfaa5f0 ===== 2020-03-30 10:24:30.710 7f6a19bf3700 1 ====== req done req=0x561d9cfd45f0 op status=0 http_status=200 latency=0.00199988s ====== 2020-03-30 10:24:30.711 7f6a193f2700 1 ====== req done req=0x561d9cfaa5f0 op status=0 http_status=200 latency=0.00299982s ====== /ceph-rgw-cnode05.rgw0.log 2020-03-30 10:07:41.309 7fb79d31c700 1 ====== req done http_status=400 ====== 2020-03-30 10:07:41.505 7fb798312700 1 ====== starting new request req=0x5565d88b45f0 ===== 2020-03-30 10:07:41.508 7fb798312700 1 ====== req done req=0x5565d88b45f0 op status=0 http_status=200 latency=0.00299982s ====== 2020-03-30 10:07:41.531 7fb79430a700 1 failed to read header: bad method 2020-03-30 10:07:41.531 7fb79430a700 1 ====== req done http_status=400 ====== 2020-03-30 10:07:41.552 7fb791304700 1 failed to read header: bad method 2020-03-30 10:07:41.552 7fb791304700 1 ====== req done http_status=400 ====== (END)

3 2

ceph cephadm generate-key => No such file or directory: '/tmp/tmp4ejhr7wh/key'
by Ml Ml 30 Mar '20

30 Mar '20

Hello List, is this a bug? root@ceph02:~# ceph cephadm generate-key Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1413, in _generate_key with open(path, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp4ejhr7wh/key' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 110, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 308, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 72, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/cephadm/module.py", line 1418, in _generate_key os.unlink(path) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp4ejhr7wh/key' root@ceph02:~# dpkg -l |grep ceph ii ceph-base 15.2.0-1~bpo10+1 amd64 common ceph daemon libraries and management tools ii ceph-common 15.2.0-1~bpo10+1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-deploy 2.0.1 all Ceph-deploy is an easy to use configuration tool ii ceph-mds 15.2.0-1~bpo10+1 amd64 metadata server for the ceph distributed file system ii ceph-mgr 15.2.0-1~bpo10+1 amd64 manager for the ceph distributed storage system ii ceph-mgr-cephadm 15.2.0-1~bpo10+1 all cephadm orchestrator module for ceph-mgr ii ceph-mgr-dashboard 15.2.0-1~bpo10+1 all dashboard module for ceph-mgr ii ceph-mgr-diskprediction-cloud 15.2.0-1~bpo10+1 all diskprediction-cloud module for ceph-mgr ii ceph-mgr-diskprediction-local 15.2.0-1~bpo10+1 all diskprediction-local module for ceph-mgr ii ceph-mgr-k8sevents 15.2.0-1~bpo10+1 all kubernetes events module for ceph-mgr ii ceph-mgr-modules-core 15.2.0-1~bpo10+1 all ceph manager modules which are always enabled ii ceph-mgr-rook 15.2.0-1~bpo10+1 all rook module for ceph-mgr ii ceph-mon 15.2.0-1~bpo10+1 amd64 monitor server for the ceph storage system ii ceph-osd 15.2.0-1~bpo10+1 amd64 OSD server for the ceph storage system ii cephadm 15.2.0-1~bpo10+1 amd64 cephadm utility to bootstrap ceph daemons with systemd and containers ii libcephfs1 10.2.11-2 amd64 Ceph distributed file system client library ii libcephfs2 15.2.0-1~bpo10+1 amd64 Ceph distributed file system client library ii python-ceph-argparse 14.2.8-1 all Python 2 utility libraries for Ceph CLI ii python3-ceph-argparse 15.2.0-1~bpo10+1 all Python 3 utility libraries for Ceph CLI ii python3-ceph-common 15.2.0-1~bpo10+1 all Python 3 utility libraries for Ceph ii python3-cephfs 15.2.0-1~bpo10+1 amd64 Python 3 libraries for the Ceph libcephfs library root@ceph02:~# cat /etc/debian_version 10.3 Thanks, Michael

2 1

Re: How do I get a sector marked bad?
by Dan van der Ster 30 Mar '20

30 Mar '20

Hi, I have a feeling that the pg repair didn't actually run yet. Sometimes if the OSDs are busy scrubbing, the repair doesn't start when you ask it to. You can force it through with something like: ceph osd set noscrub ceph osd set nodeep-scrub ceph config set osd_max_scrubs 3 ceph pg repair <the pg> ceph status # and check that the repair really started ceph config set osd_max_scrubs 1 ceph osd unset nodeep-scrub ceph osd unset noscrub Once repair runs/completes, it will rewrite the inconsistent object replica (to a new place on the disk). Check your ceph.log to see when this happens. From my experience, the PendingSectors counter will not be decremented until that sector is written again (which will happen at some random point in the future when bluestore allocates some new data there). Hope that helps, Dan On Mon, Mar 30, 2020 at 9:00 AM David Herselman <dhe(a)syrex.co> wrote: > > Hi, > > We have a single inconsistent placement group where I then subsequently triggered a deep scrub and tried doing a 'pg repair'. The placement group remains in an inconsistent state. > > How do I discard the objects for this placement group only on the one OSD and get Ceph to essentially write the data out new. Drives will only mark a sector as remapped when asked to overwrite the problematic sector or repeated reads of the failed sector eventually succeed (this is my limited understanding). > > Nothing useful in the 'ceph pg 1.35 query' output that I could decipher. Then ran 'ceph pg deep-scrub 1.35' and 'rados list-inconsistent-obj 1.35' thereafter indicates a read error on one of the copies: > {"epoch":25776,"inconsistents":[{"object":{"name":"rbd_data.746f3c94fb3a42.000000000001e48d","nspace":"","locator":"","snap":"head","version":34866184},"errors":[],"union_shard_errors":["read_error"],"selected_object_info":{"oid":{"oid":"rbd_data.746f3c94fb3a42.000000000001e48d","key":"","snapid":-2,"hash":3814100149,"max":0,"pool":1,"namespace":""},"version":"22845'1781037","prior_version":"22641'1771494","last_reqid":"client.136837683.0:124047","user_version":34866184,"size":4194304,"mtime":"2020-03-08 17:59:00.159846","local_mtime":"2020-03-08 17:59:00.159670","lost":0,"flags":["dirty","data_digest","omap_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x031cb17c","omap_digest":"0xffffffff","expected_object_size":4194304,"expected_write_size":4194304,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}},"shards":[{"osd":51,"primary":false,"errors":["read_error"],"size":4194304},{"osd":60,"primary":false,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_dig > est":"0x031cb17c"},{"osd":82,"primary":true,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_digest":"0x031cb17c"}]}]} > > /var/log/syslog: > Mar 30 08:40:40 kvm1e kernel: [74792.229021] ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0 > Mar 30 08:40:40 kvm1e kernel: [74792.230416] ata2.00: irq_stat 0x40000008 > Mar 30 08:40:40 kvm1e kernel: [74792.231715] ata2.00: failed command: READ FPDMA QUEUED > Mar 30 08:40:40 kvm1e kernel: [74792.233071] ata2.00: cmd 60/00:08:00:7a:50/04:00:c9:00:00/40 tag 1 ncq dma 524288 in > Mar 30 08:40:40 kvm1e kernel: [74792.233071] res 43/40:00:10:7b:50/00:04:c9:00:00/00 Emask 0x409 (media error) <F> > Mar 30 08:40:40 kvm1e kernel: [74792.235736] ata2.00: status: { DRDY SENSE ERR } > Mar 30 08:40:40 kvm1e kernel: [74792.237045] ata2.00: error: { UNC } > Mar 30 08:40:40 kvm1e ceph-osd[450777]: 2020-03-30 08:40:40.240 7f48a41f3700 -1 bluestore(/var/lib/ceph/osd/ceph-51) _do_read bdev-read failed: (5) Input/output error > Mar 30 08:40:40 kvm1e kernel: [74792.244914] ata2.00: configured for UDMA/133 > Mar 30 08:40:40 kvm1e kernel: [74792.244938] sd 1:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > Mar 30 08:40:40 kvm1e kernel: [74792.244942] sd 1:0:0:0: [sdb] tag#1 Sense Key : Medium Error [current] > Mar 30 08:40:40 kvm1e kernel: [74792.244945] sd 1:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed > Mar 30 08:40:40 kvm1e kernel: [74792.244949] sd 1:0:0:0: [sdb] tag#1 CDB: Read(16) 88 00 00 00 00 00 c9 50 7a 00 00 00 04 00 00 00 > Mar 30 08:40:40 kvm1e kernel: [74792.244953] blk_update_request: I/O error, dev sdb, sector 3377494800 op 0x0:(READ) flags 0x0 phys_seg 94 prio class 0 > Mar 30 08:40:40 kvm1e kernel: [74792.246238] ata2: EH complete > > > Regards > David Herselman > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Terrible IOPS performance
by Jarett DeAngelis 30 Mar '20

30 Mar '20

Hi folks, I have a three-node cluster on a 10G network with very little traffic. I have a six-OSD flash-only pool with two devices — a 1TB NVMe drive and a 256GB SATA SSD — on each node, and here’s how it benchmarks: Oof. How can I troubleshoot this? Anthony mentioned that I might be able to run more than one OSD on the NVMe — how is that done, and can I do it “on the fly” with the system already up and running like this? And, will more OSDs give me better IOPS? Thanks, Jarett

2 1

Move on cephfs not O(1)?
by Frank Schilder 30 Mar '20

30 Mar '20

Some time ago I made a surprising observation. I reorganised a directory structure and needed to move a folder one level up with a command like mv A/B/ B B contained something like 9TB in very large files. To my surprise, this command didn't return for a couple of minutes and I started to look what was going on. What I discovered was, that the mv command actually performed a full copy with a subsequent remove. I had to wait for several hours for the move to complete. I tried to reproduce this today to collect further information. However, this behaviour seems not reproducible. No matter what I try, mv completes almost instantly. I was running the original mv on mimic 13.2.2 and retried now with mimic 13.2.8. In addition, there was an OS upgrade from Centos 7.6 to 7.7. I'm using the kernel-ml versions (5.xxx). Only one cephfs mount was present at all times. My questions are: 1) Was there a change from 13.2.2 to 13.2.8 explaining this? 2) Are there (rare) conditions under which an mv on cephfs becomes a cp+rm? 3) Am I seeing ghosts? Thanks for clues and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 11

How do I get a sector marked bad?
by David Herselman 30 Mar '20

30 Mar '20

Hi, We have a single inconsistent placement group where I then subsequently triggered a deep scrub and tried doing a 'pg repair'. The placement group remains in an inconsistent state. How do I discard the objects for this placement group only on the one OSD and get Ceph to essentially write the data out new. Drives will only mark a sector as remapped when asked to overwrite the problematic sector or repeated reads of the failed sector eventually succeed (this is my limited understanding). Nothing useful in the 'ceph pg 1.35 query' output that I could decipher. Then ran 'ceph pg deep-scrub 1.35' and 'rados list-inconsistent-obj 1.35' thereafter indicates a read error on one of the copies: {"epoch":25776,"inconsistents":[{"object":{"name":"rbd_data.746f3c94fb3a42.000000000001e48d","nspace":"","locator":"","snap":"head","version":34866184},"errors":[],"union_shard_errors":["read_error"],"selected_object_info":{"oid":{"oid":"rbd_data.746f3c94fb3a42.000000000001e48d","key":"","snapid":-2,"hash":3814100149,"max":0,"pool":1,"namespace":""},"version":"22845'1781037","prior_version":"22641'1771494","last_reqid":"client.136837683.0:124047","user_version":34866184,"size":4194304,"mtime":"2020-03-08 17:59:00.159846","local_mtime":"2020-03-08 17:59:00.159670","lost":0,"flags":["dirty","data_digest","omap_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x031cb17c","omap_digest":"0xffffffff","expected_object_size":4194304,"expected_write_size":4194304,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}},"shards":[{"osd":51,"primary":false,"errors":["read_error"],"size":4194304},{"osd":60,"primary":false,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_digest":"0x031cb17c"},{"osd":82,"primary":true,"errors":[],"size":4194304,"omap_digest":"0xffffffff","data_digest":"0x031cb17c"}]}]} /var/log/syslog: Mar 30 08:40:40 kvm1e kernel: [74792.229021] ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x0 Mar 30 08:40:40 kvm1e kernel: [74792.230416] ata2.00: irq_stat 0x40000008 Mar 30 08:40:40 kvm1e kernel: [74792.231715] ata2.00: failed command: READ FPDMA QUEUED Mar 30 08:40:40 kvm1e kernel: [74792.233071] ata2.00: cmd 60/00:08:00:7a:50/04:00:c9:00:00/40 tag 1 ncq dma 524288 in Mar 30 08:40:40 kvm1e kernel: [74792.233071] res 43/40:00:10:7b:50/00:04:c9:00:00/00 Emask 0x409 (media error) <F> Mar 30 08:40:40 kvm1e kernel: [74792.235736] ata2.00: status: { DRDY SENSE ERR } Mar 30 08:40:40 kvm1e kernel: [74792.237045] ata2.00: error: { UNC } Mar 30 08:40:40 kvm1e ceph-osd[450777]: 2020-03-30 08:40:40.240 7f48a41f3700 -1 bluestore(/var/lib/ceph/osd/ceph-51) _do_read bdev-read failed: (5) Input/output error Mar 30 08:40:40 kvm1e kernel: [74792.244914] ata2.00: configured for UDMA/133 Mar 30 08:40:40 kvm1e kernel: [74792.244938] sd 1:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Mar 30 08:40:40 kvm1e kernel: [74792.244942] sd 1:0:0:0: [sdb] tag#1 Sense Key : Medium Error [current] Mar 30 08:40:40 kvm1e kernel: [74792.244945] sd 1:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed Mar 30 08:40:40 kvm1e kernel: [74792.244949] sd 1:0:0:0: [sdb] tag#1 CDB: Read(16) 88 00 00 00 00 00 c9 50 7a 00 00 00 04 00 00 00 Mar 30 08:40:40 kvm1e kernel: [74792.244953] blk_update_request: I/O error, dev sdb, sector 3377494800 op 0x0:(READ) flags 0x0 phys_seg 94 prio class 0 Mar 30 08:40:40 kvm1e kernel: [74792.246238] ata2: EH complete Regards David Herselman

1 0

BlueStore and checksum
by Priya Sehgal 30 Mar '20

30 Mar '20

Hi, I am trying to find out whether Ceph has a method to detect silent corruption such as bit rot. I came across this text in a book - "Mastering Ceph : Infrastructure Storage Solution with the latest Ceph release" by Nick Fisk - Luminous release of Ceph employs ZFS-like ability to checksum data at every read. BlueStore calculates and stores the crc32 checksum of any data that is written. On each read request, BlueStore reads the checksum and compares it with the data read from the device. If there is a mismatch, BlueStore will report read error and repair damage. Ceph will then retry the read from another OSD holding the object. I have two questions: 1. If there is a mismatch will the ceph user (who initiated GET) receive any error message and he has to retry OR will this error be auto-corrected by Ceph and the data would be returned from the other OSD (user will never know what went wrong underneath? 2. Does this feature work for Multi-part upload objects also? -- Regards, Priya

2 2

Ceph Trace System
by Liu, Changcheng 29 Mar '20

29 Mar '20

Hi all, Does anyone know how to trace the write/read request from client to rbd/rados/OSD? Is there any useful document besides the doc/dev/blkin? I'm wondering how to trace the large distributed storage system e.g. Ceph to observer/monitor it. B.R. Changcheng

1 0

Re: samba ceph-vfs and scrubbing interval
by Marco Savoca 28 Mar '20

28 Mar '20

Hi Jeff, i have digged a bit deeper and I was able to get write access on the ceph-vfs share. But the file and directory permissions are a mess. It seems that ceph-VfL does not evaluate the secondary group permissions. > Am 27.03.2020 um 18:14 schrieb Dr. Marco Savoca <quaternionma(a)gmail.com>: > > > „No. I haven't tested it in some time, but it does allow clients to > write. When you say you can't get write access, what are you doing to > test this, and what error are you getting back?“ > > For example: after successfull connection via smbclient it says „NT_STATUS_ACCESS_DENIED making remote Directory“ or > > „NT_STATUS_ACCESS_DENIED deleting remote file“ when I try to make a directory or to delete a file. If the same samba user connects to a share with a mounted path, everything works as expeted, so that there should not be some ACL Errors. > > > > > > Von: Jeff Layton > Gesendet: Freitag, 27. März 2020 13:05 > An: Marco Savoca; ceph-users(a)ceph.io > Betreff: Re: [ceph-users] samba ceph-vfs and scrubbing interval > > On Fri, 2020-03-27 at 12:00 +0100, Marco Savoca wrote: > > Hi all, > > > > i‘m running a 3 node ceph cluster setup with collocated mons and mds > > for actually 3 filesystems at home since mimic. I’m planning to > > downgrade to one FS and use RBD in the future, but this is another > > story. I’m using the cluster as cold storage on spindles with EC-pools > > for archive purposes. The cluster usually does not run 24/7. I > > actually managed to upgrade to octopus without problems yesterday. So > > first of all: great job with the release. > > > > Now I have a little problem and a general question to address. > > > > I have tried to share the CephFS via samba and the ceph-vfs module but > > I could not manage to get write access (read access is not a problem) > > to the share (even with the admin key). When I share the mounted path > > (kernel module or fuser mount) instead as usual there are no problems > > at all. Is ceph-vfs generally read only and I missed this point? > > No. I haven't tested it in some time, but it does allow clients to > write. When you say you can't get write access, what are you doing to > test this, and what error are you getting back? > > > Furthermore I suppose, that there is no possibility to choose between > > the different mds namespaces, right? > > > > Yeah, doesn't look like anyone has added that. That would probably be > pretty easy to add, though it would take a little while to trickle out > to the distros. > > > Now the general question. Since the cluster does not run 24/7 as > > stated and is turned on perhaps once a week for a couple of hours on > > demand, what are reasonable settings for the scrubbing intervals? As I > > said, the storage is cold and there is mostly read i/o. The archiving > > process adds approximately 0.5 % of new data of the cluster’s total > > storage capacity. > > -- > Jeff Layton <jlayton(a)redhat.com> > >

1 0

Leave of absence...
by Sage Weil 28 Mar '20

28 Mar '20

Hi everyone, I am taking time off from the Ceph project and from Red Hat, starting in April and extending through the US election in November. I will initially be working with an organization focused on voter registration and turnout and combating voter suppression and disinformation campaigns. During this time I will maintain some involvement in the Ceph community, primarily around strategic planning for Pacific and the Ceph Foundation, but most of my time will be focused elsewhere. Most decision making around Ceph will remain in the capable hands of the Ceph Leadership Team and component leads--I have the utmost confidence in their judgement and abilities. Yehuda Sadeh and Josh Durgin will be filling in to provide high-level guidance where needed. I’ll be participating in the Pacific planning meetings planned for next week, which will be important in kicking off development for Pacific: https://ceph.io/cds/ceph-developer-summit-pacific/ I am extremely proud of what we have accomplished with the Octopus release, and I believe the Ceph community will continue to do great things with Pacific! I look forward to returning at the end of the year to help wrap up the release and (hopefully) get things ready for Cephalocon next March. Most of all, I am excited to become engaged in another effort that I feel strongly about--one that will have a very real impact on my kids’ futures--and that will be easier to explain to lay people! :) Thanks! sage

2 1

Space leak in Bluestore
by vitalif＠yourcmc.ru 27 Mar '20

27 Mar '20

Hi. I'm experiencing some kind of a space leak in Bluestore. I use EC, compression and snapshots. First I thought that the leak was caused by "virtual clones" (issue #38184). However, then I got rid of most of the snapshots, but continued to experience the problem. I suspected something when I added a new disk to the cluster and free space in the cluster didn't increase (!). So to track down the issue I moved one PG (34.1a) using upmaps from osd11,6,0 to osd6,0,7 and then back to osd11,6,0. It ate +59 GB after the first move and +51 GB after the second. As I understand this proves that it's not #38184. Devirtualizaton of virtual clones couldn't eat additional space after SECOND rebalance of the same PG. The PG has ~39000 objects, it is EC 2+1 and the compression is enabled. Compression ratio is about ~2.7 in my setup, so the PG should use ~90 GB raw space. Before and after moving the PG I stopped osd0, mounted it with ceph-objectstore-tool with debug bluestore = 20/20 and opened the 34.1a***/all directory. It seems to dump all object extents into the log in that case. So now I have two logs with all allocated extents for osd0 (I hope all extents are there). I parsed both logs and added all compressed blob sizes together ("get_ref Blob ... 0x20000 -> 0x... compressed"). But they add up to ~39 GB before first rebalance (34.1as2), ~22 GB after it (34.1as1) and ~41 GB again after the second move (34.1as2) which doesn't indicate a leak. But the raw space usage still exceeds initial by a lot. So it's clear that there's a leak somewhere. What additional details can I provide for you to identify the bug? I posted the same message in the issue tracker, https://tracker.ceph.com/issues/44731 -- Vitaliy Filippov

5 9

Identify slow ops
by Thomas Schneider 27 Mar '20

27 Mar '20

Hi, the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing. root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d) data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops There's no error on services (mgr, mon, osd). Can you please advise how to identify the root cause of this slow ops? THX

3 6

Help: corrupt pg
by Jake Grimmett 27 Mar '20

27 Mar '20

Dear All, We are "in a bit of a pickle"... No reply to my message (23/03/2020), subject "OSD: FAILED ceph_assert(clone_size.count(clone))" So I'm presuming it's not possible to recover the crashed OSD This is bad news, as one pg may be lost, (we are using EC 8+2, pg dump shows [NONE,NONE,NONE,388,125,25,427,226,77,154] ) Without this pg we have 1.8PB of broken cephfs. I could rebuild the cluster from scratch, but this means no user backups for a couple of weeks. The cluster has 10 nodes, uses an EC 8:2 pool for cephfs data (replicated NVMe metdata pool) and is running Nautilus 14.2.8 Clearly, it would be nicer if we could fix the OSD, but if this isn't possible, can someone confirm that the right procedure to recover from a corrupt pg is: 1) Stop all client access 2) find all files that store data on the bad pg, with: # cephfs-data-scan pg_files /backup 5.750 2> /dev/null > /root/bad_files 3) delete all of these bad files - presumably using truncate? or is "rm" fine? 4) destroy the bad pg # ceph osd force-create-pg 5.750 5) Copy the missing files back with rsync or similar... a better "recipe" or other advice gratefully received, best regards, Jake **** Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

3 4

fast luminous -> nautilus -> octopus upgrade could lead to assertion failure on OSD
by kefu chai 27 Mar '20

27 Mar '20

hi folks, if you are upgrading from luminous to octopus, or you plan to do so, please read on. in octopus, osd will crash if it processes an osdmap whose require_osd_release flag is still luminous. this only happens if a cluster upgrades very quickly from luminous to nautilus and to octopus. in this case, there are good chances that an octopus OSD will need to consume osdmaps which were created back in luminous. because we assumed that ceph did not ugprade across major releases, in octopus, OSD will panic at seeing osdmap from luminous. this is a known bug[0], and already fixed in master. and the next octopus release will include the fix to address this issue. as a workaround, you need to wait a while after running ceph osd require-osd-release nautilus and optionally inject lots of osdmaps into cluster to ensure that the old luminous osd maps are trimmed: for i in `seq 500`; do ceph osd blacklist add 192.168.0.1 ceph osd blacklist rm 192.168.0.1 done after the whole cluster are active+clean, then upgrade to octopus. happy upgrading! cheers, -- [0] https://tracker.ceph.com/issues/44759 -- Regards Kefu Chai

1 0

[ceph][nautilus] error initalizing secondary zone
by Ignazio Cassano 27 Mar '20

27 Mar '20

Hello , I am trying to initializing the secondary zone pulling the realm define in the primary zone: radosgw-admin realm pull --rgw-realm=nivola --url=http://10.102.184.190:8080 --access-key=access --secret=secret The following errors appears: request failed: (16) Device or resource busy Could you help me please ? Ignazio

1 1

Re: Combining erasure coding and replication?
by Eugen Block 27 Mar '20

27 Mar '20

Hi Brett, > Our concern with Ceph is the cost of having three replicas. Storage > may be cheap but I’d rather not buy ANOTHER 5pb for a third replica > if there are ways to do this more efficiently. Site-level redundancy > is important to us so we can’t simply create an erasure-coded volume > across two buildings – if we lose power to a building, the entire > array would become unavailable. can you elaborate on that? Why is EC not an option? We have installed several clusters with two datacenters resilient to losing a whole dc (and additional disks if required). So it's basically the choice of the right EC profile. Or did I misunderstand something? Zitat von Brett Randall <brett.randall(a)gmail.com>: > Hi all > > Had a fun time trying to join this list, hopefully you don’t get > this message 3 times! > > On to Ceph… We are looking at setting up our first ever Ceph cluster > to replace Gluster as our media asset storage and production system. > The Ceph cluster will have 5pb of usable storage. Whether we use it > as object-storage, or put CephFS in front of it, is still TBD. > > Obviously we’re keen to protect this data well. Our current Gluster > setup utilises RAID-6 on each of the nodes and then we have a single > replica of each brick. The Gluster bricks are split between > buildings so that the replica is guaranteed to be in another > premises. By doing it this way, we guarantee that we can have a > decent number of disk or node failures (even an entire building) > before we lose both connectivity and data. > > Our concern with Ceph is the cost of having three replicas. Storage > may be cheap but I’d rather not buy ANOTHER 5pb for a third replica > if there are ways to do this more efficiently. Site-level redundancy > is important to us so we can’t simply create an erasure-coded volume > across two buildings – if we lose power to a building, the entire > array would become unavailable. Likewise, we can’t simply have a > single replica – our fault tolerance would drop way down on what it > is right now. > > Is there a way to use both erasure coding AND replication at the > same time in Ceph to mimic the architecture we currently have in > Gluster? I know we COULD just create RAID6 volumes on each node and > use the entire volume as a single OSD, but that this is not the > recommended way to use Ceph. So is there some other way? > > Apologies if this is a nonsensical question, I’m still trying to > wrap my head around Ceph, CRUSH maps, placement rules, volume types, > etc etc! > > TIA > > Brett > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 3

Re: Combining erasure coding and replication?
by Lars Täuber 27 Mar '20

27 Mar '20

Hi Brett, I'm far from being an expert, but you may consider rbd-mirroring between EC-pools. Cheers, Lars Am Fri, 27 Mar 2020 06:28:02 +0000 schrieb Brett Randall <brett.randall(a)gmail.com>: > Hi all > > Had a fun time trying to join this list, hopefully you don’t get this message 3 times! > > On to Ceph… We are looking at setting up our first ever Ceph cluster to replace Gluster as our media asset storage and production system. The Ceph cluster will have 5pb of usable storage. Whether we use it as object-storage, or put CephFS in front of it, is still TBD. > > Obviously we’re keen to protect this data well. Our current Gluster setup utilises RAID-6 on each of the nodes and then we have a single replica of each brick. The Gluster bricks are split between buildings so that the replica is guaranteed to be in another premises. By doing it this way, we guarantee that we can have a decent number of disk or node failures (even an entire building) before we lose both connectivity and data. > > Our concern with Ceph is the cost of having three replicas. Storage may be cheap but I’d rather not buy ANOTHER 5pb for a third replica if there are ways to do this more efficiently. Site-level redundancy is important to us so we can’t simply create an erasure-coded volume across two buildings – if we lose power to a building, the entire array would become unavailable. Likewise, we can’t simply have a single replica – our fault tolerance would drop way down on what it is right now. > > Is there a way to use both erasure coding AND replication at the same time in Ceph to mimic the architecture we currently have in Gluster? I know we COULD just create RAID6 volumes on each node and use the entire volume as a single OSD, but that this is not the recommended way to use Ceph. So is there some other way? > > Apologies if this is a nonsensical question, I’m still trying to wrap my head around Ceph, CRUSH maps, placement rules, volume types, etc etc! > > TIA > > Brett > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de

1 0

Combining erasure coding and replication?
by Brett Randall 27 Mar '20

27 Mar '20

Hi all Had a fun time trying to join this list, hopefully you don’t get this message 3 times! On to Ceph… We are looking at setting up our first ever Ceph cluster to replace Gluster as our media asset storage and production system. The Ceph cluster will have 5pb of usable storage. Whether we use it as object-storage, or put CephFS in front of it, is still TBD. Obviously we’re keen to protect this data well. Our current Gluster setup utilises RAID-6 on each of the nodes and then we have a single replica of each brick. The Gluster bricks are split between buildings so that the replica is guaranteed to be in another premises. By doing it this way, we guarantee that we can have a decent number of disk or node failures (even an entire building) before we lose both connectivity and data. Our concern with Ceph is the cost of having three replicas. Storage may be cheap but I’d rather not buy ANOTHER 5pb for a third replica if there are ways to do this more efficiently. Site-level redundancy is important to us so we can’t simply create an erasure-coded volume across two buildings – if we lose power to a building, the entire array would become unavailable. Likewise, we can’t simply have a single replica – our fault tolerance would drop way down on what it is right now. Is there a way to use both erasure coding AND replication at the same time in Ceph to mimic the architecture we currently have in Gluster? I know we COULD just create RAID6 volumes on each node and use the entire volume as a single OSD, but that this is not the recommended way to use Ceph. So is there some other way? Apologies if this is a nonsensical question, I’m still trying to wrap my head around Ceph, CRUSH maps, placement rules, volume types, etc etc! TIA Brett

1 0

Combining erasure coding and replication?
by Brett Randall 27 Mar '20

27 Mar '20

Hi all We are looking at setting up our first ever Ceph cluster to replace Gluster as our media asset storage and production system. The Ceph cluster will have 5pb of usable storage. Whether we use it as object-storage, or put CephFS in front of it, is still TBD. Obviously we’re keen to protect this data well. Our current Gluster setup utilises RAID-6 on each of the nodes and then we have a single replica of each brick. The Gluster bricks are split between buildings so that the replica is guaranteed to be in another premises. By doing it this way, we guarantee that we can have a decent number of disk or node failures (even an entire building) before we lose both connectivity and data. Our concern with Ceph is the cost of having three replicas. Storage may be cheap but I’d rather not buy ANOTHER 5pb for a third replica if there are ways to do this more efficiently. Site-level redundancy is important to us so we can’t simply create an erasure-coded volume across two buildings – if we lose power to a building, the entire array would become unavailable. Likewise, we can’t simply have a single replica – our fault tolerance would drop way down on what it is right now. Is there a way to use both erasure coding AND replication at the same time in Ceph to mimic the architecture we currently have in Gluster? I know we COULD just create RAID6 volumes on each node and use the entire volume as a single OSD, but that this is not the recommended way to use Ceph. So is there some other way? Apologies if this is a nonsensical question, I’m still trying to wrap my head around Ceph, CRUSH maps, placement rules, volume types, etc etc! TIA Brett

1 0

Ceph rbd mirror and object storage multisite
by Ignazio Cassano 26 Mar '20

26 Mar '20

Hello All, I am going to test rbd mirroring and object storage multisite. I would like to know which network is used by rbd mirror ( ceph public or cluster network ?) Same question for object storage multisite.... What about firewall ? What about bandwidth ? Our sites are connected with 1Gbs network ? How many volumes can I mirror in this scenario? Could anyone help me ? Best Regards Ignazio

1 0

Performance characteristics of ‘if-none-match’ on rgw
by akmd＠metaswitch.com 26 Mar '20

26 Mar '20

Hello, I am observing non-intuitive results for a performance test using the S3 API to RGW. I am wondering if others have similar experiences or knowledge here. Our application is using the “if-none-match” header on S3-API requests. This header is set by the application if it already has a copy of the object in question but wishes to check if there is a newer version. If the etag of the current object matches then RGW sends a 304 response, and if it doesn’t it sends the updated content of the object. We’re observing that the response time of requests resulting in “304 Not Modified” is typically slower than those for normal object retrieval. This wasn’t intuitive to me – in the 304 case there is no content to transfer over the network and I would expect the request can be satisfied just by looking at the RGW index (I was under the impression that metadata including etag is in the index). Anecdotally, HEAD requests see similar results but I haven't yet analysed in full. Does anyone else have data or experience about expected performance of this scenario? Are there any potential avenues for optimization of configuration ? What kind of commands can I use to debug this further ? Some details of the current setup: => ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) => Objects are typically 80-100KB. => Versioning is enabled on the bucket. => Our requests specify a Range header (hence will generate 206 not 200). => Multisite features are enabled. => Bucket has 20 shards – I’ve put a dump of "bucket limits" below. Performance results Response, Request Count, Median, 75th percentile, 90th percentile, 95th percentile, 206 Partial, 20473, 3, 3, 16, 129, 1200 304 Not Modified, 15644, 9, 16, 46, 212, 1192 Bucket details { "bucket": "albansstack-scsdata", "tenant": "", "num_objects": 465780, "num_shards": 20, "objects_per_shard": 23289, "fill_status": "OK" }, Many thanks, Alistair.

1 0

octopus upgrade stuck: Assertion `map->require_osd_release >= ceph_release_t::mimic' failed.
by Ml Ml 26 Mar '20

26 Mar '20

Hello List, i followed: https://ceph.io/releases/v15-2-0-octopus-released/ I came from a healthy nautilus and i am stuck at: 5.) Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD host When i try to start an osd like this, i get: /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph ... 2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223 log_to_monitors {default=true} ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. *** Caught signal (Aborted) ** in thread 7f274854f700 thread_name:tp_osd_tp Aborted My current status: root@ceph03:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 60.70999 root default -2 20.25140 host ceph01 0 hdd 1.71089 osd.0 up 1.00000 1.00000 8 hdd 2.67029 osd.8 up 1.00000 1.00000 11 hdd 1.59999 osd.11 up 1.00000 1.00000 12 hdd 1.59999 osd.12 up 1.00000 1.00000 14 hdd 2.79999 osd.14 up 1.00000 1.00000 18 hdd 1.59999 osd.18 up 1.00000 1.00000 22 hdd 2.79999 osd.22 up 1.00000 1.00000 23 hdd 2.79999 osd.23 up 1.00000 1.00000 26 hdd 2.67029 osd.26 up 1.00000 1.00000 -3 23.05193 host ceph02 2 hdd 2.67029 osd.2 up 1.00000 1.00000 3 hdd 2.00000 osd.3 up 1.00000 1.00000 7 hdd 2.67029 osd.7 up 1.00000 1.00000 9 hdd 2.67029 osd.9 up 1.00000 1.00000 13 hdd 2.00000 osd.13 up 1.00000 1.00000 16 hdd 1.59999 osd.16 up 1.00000 1.00000 19 hdd 2.38409 osd.19 up 1.00000 1.00000 24 hdd 2.67020 osd.24 up 1.00000 1.00000 25 hdd 1.71649 osd.25 up 1.00000 1.00000 28 hdd 2.67029 osd.28 up 1.00000 1.00000 -4 17.40666 host ceph03 5 hdd 1.71660 osd.5 down 1.00000 1.00000 6 hdd 1.71660 osd.6 down 1.00000 1.00000 10 hdd 2.67029 osd.10 down 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000 root@ceph03:~# ceph osd dump | grep require_osd_release require_osd_release nautilus root@ceph03:~# ceph osd versions { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 } oot@ceph03:~# ceph mon dump | grep min_mon_release dumped monmap epoch 12 min_mon_release 15 (octopus) ceph versions { "mon": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "mgr": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "osd": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 }, "mds": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "overall": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19, "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9 } } Why does it complain about map->require_osd_release >= ceph_release_t::mimic ? Cheers, Michael

2 2

14.2.7 MDS High Virtual Memory
by Dave Hall 26 Mar '20

26 Mar '20

Hello. I have a cluster with 3 MDSs - 2 active and 1 standby. Looking at 'top' I see that the 2 active MDSs have relatively low resident memory - less than 3 g, but very high virtual memory - 21 g and 41 g. Is it safe to just restart these processes one at a time to release the virtual memory? Also, is there something that needs to be tweaked to prevent the MDSs from accumulating so much memory? Thanks. -Dave -- Dave Hall Binghamton University

2 1

Re: Exporting
by Eugen Block 26 Mar '20

26 Mar '20

Hi, have you tried to create a snapshot of that image and export the snapshot? Or maybe clone the snapshot first and then export it? Does it also fail? Zitat von Rhian Resnick <rresnick(a)fau.edu>: > Evening, > > We are running into issues exporting a disk image from ceph rbd. > When we attempt to export an rbd image in a cache tiered > erasure-coded pool on Luminus. > > All the other disks are working fine but this one is acting up. We > have a bit of important data on other disks so obviously want to > make sure this doesn't happen to those. > > [root@ceph-p-mon1 home]# rbd export one/one-177-588-0 one-177-588-0 > Exporting image: 8% complete...rbd: error reading from source image > at offset 5456789504: (5) Input/output error > 2020-03-23 20:11:29.210718 7f2f3effd700 -1 > librbd::io::ObjectRequest: 0x7f2f2c128f90 handle_read_object: failed > to read from object: (5) Input/output error > 2020-03-23 20:11:29.565184 7f2f3e7fc700 -1 > librbd::io::ObjectRequest: 0x7f2f280c84d0 handle_read_cache: failed > to read from cache: (5) Input/output error > Exporting image: 8% complete...failed. > rbd: export error: (5) Input/output error > > > Any thoughts would be appreciated. > > > Some info: > > [root@ceph-p-mon1 home]# rbd info one/one-177-588-0 > rbd image 'one-177-588-0': > size 58.6GiB in 15000 objects > order 22 (4MiB objects) > block_name_prefix: rbd_data.84a01279e2a9e3 > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, deep-flatten > flags: > create_timestamp: Fri Apr 20 17:06:09 2018 > parent: one/one-177@snap > overlap: 2.20GiB > > [root@ceph-p-mon1 home]# ceph status > cluster: > id: 6a2e8f21-bca2-492b-8869-eecc995216cc > health: HEALTH_OK > > services: > mon: 3 daemons, quorum ceph-p-mon2,ceph-p-mon1,ceph-p-mon3 > mgr: ceph-p-mon2(active) > mds: cephfsec-1/1/1 up {0=ceph-p-mon2=up:active}, 6 up:standby > osd: 155 osds: 154 up, 154 in > > data: > pools: 6 pools, 5904 pgs > objects: 145.53M objects, 192TiB > usage: 253TiB used, 290TiB / 543TiB avail > pgs: 5896 active+clean > 8 active+clean+scrubbing+deep > > io: > client: 921KiB/s rd, 5.68MiB/s wr, 110op/s rd, 29op/s wr > cache: 5.65MiB/s flush, 0op/s promote > > > [root@ceph-p-mon1 home]# rpm -qa | grep ceph > ceph-common-12.2.9-0.el7.x86_64 > ceph-mds-12.2.9-0.el7.x86_64 > ceph-radosgw-12.2.9-0.el7.x86_64 > ceph-mgr-12.2.9-0.el7.x86_64 > ceph-12.2.9-0.el7.x86_64 > collectd-ceph-5.8.1-1.el7.x86_64 > ceph-deploy-2.0.1-0.noarch > libcephfs2-12.2.9-0.el7.x86_64 > python-cephfs-12.2.9-0.el7.x86_64 > ceph-selinux-12.2.9-0.el7.x86_64 > ceph-osd-12.2.9-0.el7.x86_64 > ceph-base-12.2.9-0.el7.x86_64 > ceph-mon-12.2.9-0.el7.x86_64 > ceph-release-1-1.el7.noarch > > > Rhian Resnick > > Associate Director Research Computing > > Enterprise Systems > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus
by Ml Ml 25 Mar '20

25 Mar '20

i brought them up manually. Decided to upgrade to octopus, but i am stuck there now. I will open a new thread for it. On Wed, Mar 25, 2020 at 6:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Still down? > > > -----Original Message----- > Cc: ceph-users > Subject: [ceph-users] Re: OSDs wont mount on Debian 10 (Buster) with > Nautilus > > What does the osd error log say? I already have bluestore, if you have > file store maybe you should inspect the mounted fs of /dev/sdd2 eg. > maybe there permissions need to be changed. But first check the errors > of one osd. > > ( You did you reset the failed service with somethig like this systemctl > reset-failed ceph-osd@0 not? ) > > > -----Original Message----- > Cc: ceph-users > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > Still no luck. > > But the working OSDs have no partition: > OSD.1 => /dev/sdj > OSD.5 => /dev/sdb > OSD.6 => /dev/sdbc > OSD.10 => /dev/sdl > > > Where as the rest has: > > root@ceph03:~# ls -l /dev/sd* > brw-rw---- 1 root disk 8, 0 Mar 25 16:23 /dev/sda > brw-rw---- 1 root disk 8, 1 Mar 25 16:23 /dev/sda1 > brw-rw---- 1 root disk 8, 2 Mar 25 16:23 /dev/sda2 > brw-rw---- 1 root disk 8, 3 Mar 25 16:23 /dev/sda3 > brw-rw---- 1 root disk 8, 16 Mar 25 16:23 /dev/sdb > brw-rw---- 1 root disk 8, 32 Mar 25 16:23 /dev/sdc > brw-rw---- 1 root disk 8, 48 Mar 25 16:23 /dev/sdd > brw-rw---- 1 root disk 8, 49 Mar 25 16:23 /dev/sdd1 > brw-rw---- 1 ceph ceph 8, 50 Mar 25 16:23 /dev/sdd2 > brw-rw---- 1 root disk 8, 64 Mar 25 16:23 /dev/sde > brw-rw---- 1 root disk 8, 65 Mar 25 16:23 /dev/sde1 > brw-rw---- 1 ceph ceph 8, 66 Mar 25 16:23 /dev/sde2 > brw-rw---- 1 root disk 8, 80 Mar 25 16:23 /dev/sdf > brw-rw---- 1 root disk 8, 81 Mar 25 16:23 /dev/sdf1 > brw-rw---- 1 ceph ceph 8, 82 Mar 25 16:23 /dev/sdf2 > brw-rw---- 1 root disk 8, 96 Mar 25 16:23 /dev/sdg > brw-rw---- 1 root disk 8, 97 Mar 25 16:23 /dev/sdg1 > brw-rw---- 1 ceph ceph 8, 98 Mar 25 16:23 /dev/sdg2 > brw-rw---- 1 root disk 8, 112 Mar 25 16:23 /dev/sdh > brw-rw---- 1 root disk 8, 113 Mar 25 16:23 /dev/sdh1 > brw-rw---- 1 ceph ceph 8, 114 Mar 25 16:23 /dev/sdh2 > brw-rw---- 1 root disk 8, 128 Mar 25 16:23 /dev/sdi > brw-rw---- 1 root disk 8, 129 Mar 25 16:23 /dev/sdi1 > brw-rw---- 1 ceph ceph 8, 130 Mar 25 16:23 /dev/sdi2 > brw-rw---- 1 root disk 8, 144 Mar 25 16:23 /dev/sdj > brw-rw---- 1 root disk 8, 160 Mar 25 16:23 /dev/sdk > brw-rw---- 1 root disk 8, 161 Mar 25 16:23 /dev/sdk1 > brw-rw---- 1 ceph ceph 8, 162 Mar 25 16:23 /dev/sdk2 > brw-rw---- 1 root disk 8, 176 Mar 25 16:23 /dev/sdl > brw-rw---- 1 root disk 8, 192 Mar 25 16:23 /dev/sdm > brw-rw---- 1 root disk 8, 193 Mar 25 16:23 /dev/sdm1 > brw-rw---- 1 ceph ceph 8, 194 Mar 25 16:23 /dev/sdm2 > > Did i miss to convert to bluestore or something? > > > On Wed, Mar 25, 2020 at 4:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > Try this > > > > chown ceph.ceph /dev/sdc2 > > chown ceph.ceph /dev/sdd2 > > chown ceph.ceph /dev/sde2 > > chown ceph.ceph /dev/sdf2 > > chown ceph.ceph /dev/sdg2 > > chown ceph.ceph /dev/sdh2 > > > > > > > > -----Original Message----- > > From: Ml Ml [mailto:mliebherr99@googlemail.com] > > Sent: 25 March 2020 16:22 > > To: Marc Roos > > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > > Nautilus > > > > They where indeed disabled. I enabled them with: > > > > systemctl enable ceph-osd@4 > > systemctl enable ceph-osd@15 > > systemctl enable ceph-osd@17 > > systemctl enable ceph-osd@20 > > systemctl enable ceph-osd@21 > > systemctl enable ceph-osd@27 > > systemctl enable ceph-osd@32 > > > > But they still wont start. > > > > On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > > wrote: > > > > > > > > > I had something similar. My osd were disabled, maybe this installer > > > of > > > > > nautilus does that check > > > > > > systemctl is-enabled ceph-osd@0 > > > > > > https://tracker.ceph.com/issues/44102 > > > >

2 1

Luminous upgrade question
by Shain Miley 25 Mar '20

25 Mar '20

Hi, We are thinking about upgrading our cluster currently running ceph version 12.2.12. I am wondering if we should be looking at upgrading to the latest version of Mimic or the latest version Nautilus. Can anyone here please provide a suggestion…I continue to be a little bit confused about the correct Ceph upgrade path given all the various versions that are out there currently. Thank you in advance. Shain

3 2

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus
by Ml Ml 25 Mar '20

25 Mar '20

Still no luck. But the working OSDs have no partition: OSD.1 => /dev/sdj OSD.5 => /dev/sdb OSD.6 => /dev/sdbc OSD.10 => /dev/sdl Where as the rest has: root@ceph03:~# ls -l /dev/sd* brw-rw---- 1 root disk 8, 0 Mar 25 16:23 /dev/sda brw-rw---- 1 root disk 8, 1 Mar 25 16:23 /dev/sda1 brw-rw---- 1 root disk 8, 2 Mar 25 16:23 /dev/sda2 brw-rw---- 1 root disk 8, 3 Mar 25 16:23 /dev/sda3 brw-rw---- 1 root disk 8, 16 Mar 25 16:23 /dev/sdb brw-rw---- 1 root disk 8, 32 Mar 25 16:23 /dev/sdc brw-rw---- 1 root disk 8, 48 Mar 25 16:23 /dev/sdd brw-rw---- 1 root disk 8, 49 Mar 25 16:23 /dev/sdd1 brw-rw---- 1 ceph ceph 8, 50 Mar 25 16:23 /dev/sdd2 brw-rw---- 1 root disk 8, 64 Mar 25 16:23 /dev/sde brw-rw---- 1 root disk 8, 65 Mar 25 16:23 /dev/sde1 brw-rw---- 1 ceph ceph 8, 66 Mar 25 16:23 /dev/sde2 brw-rw---- 1 root disk 8, 80 Mar 25 16:23 /dev/sdf brw-rw---- 1 root disk 8, 81 Mar 25 16:23 /dev/sdf1 brw-rw---- 1 ceph ceph 8, 82 Mar 25 16:23 /dev/sdf2 brw-rw---- 1 root disk 8, 96 Mar 25 16:23 /dev/sdg brw-rw---- 1 root disk 8, 97 Mar 25 16:23 /dev/sdg1 brw-rw---- 1 ceph ceph 8, 98 Mar 25 16:23 /dev/sdg2 brw-rw---- 1 root disk 8, 112 Mar 25 16:23 /dev/sdh brw-rw---- 1 root disk 8, 113 Mar 25 16:23 /dev/sdh1 brw-rw---- 1 ceph ceph 8, 114 Mar 25 16:23 /dev/sdh2 brw-rw---- 1 root disk 8, 128 Mar 25 16:23 /dev/sdi brw-rw---- 1 root disk 8, 129 Mar 25 16:23 /dev/sdi1 brw-rw---- 1 ceph ceph 8, 130 Mar 25 16:23 /dev/sdi2 brw-rw---- 1 root disk 8, 144 Mar 25 16:23 /dev/sdj brw-rw---- 1 root disk 8, 160 Mar 25 16:23 /dev/sdk brw-rw---- 1 root disk 8, 161 Mar 25 16:23 /dev/sdk1 brw-rw---- 1 ceph ceph 8, 162 Mar 25 16:23 /dev/sdk2 brw-rw---- 1 root disk 8, 176 Mar 25 16:23 /dev/sdl brw-rw---- 1 root disk 8, 192 Mar 25 16:23 /dev/sdm brw-rw---- 1 root disk 8, 193 Mar 25 16:23 /dev/sdm1 brw-rw---- 1 ceph ceph 8, 194 Mar 25 16:23 /dev/sdm2 Did i miss to convert to bluestore or something? On Wed, Mar 25, 2020 at 4:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Try this > > chown ceph.ceph /dev/sdc2 > chown ceph.ceph /dev/sdd2 > chown ceph.ceph /dev/sde2 > chown ceph.ceph /dev/sdf2 > chown ceph.ceph /dev/sdg2 > chown ceph.ceph /dev/sdh2 > > > > -----Original Message----- > From: Ml Ml [mailto:mliebherr99@googlemail.com] > Sent: 25 March 2020 16:22 > To: Marc Roos > Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > They where indeed disabled. I enabled them with: > > systemctl enable ceph-osd@4 > systemctl enable ceph-osd@15 > systemctl enable ceph-osd@17 > systemctl enable ceph-osd@20 > systemctl enable ceph-osd@21 > systemctl enable ceph-osd@27 > systemctl enable ceph-osd@32 > > But they still wont start. > > On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> > wrote: > > > > > > I had something similar. My osd were disabled, maybe this installer of > > > nautilus does that check > > > > systemctl is-enabled ceph-osd@0 > > > > https://tracker.ceph.com/issues/44102 > > > > > > > > > > -----Original Message----- > > From: Ml Ml [mailto:mliebherr99@googlemail.com] > > Sent: 25 March 2020 16:05 > > To: ceph-users > > Subject: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > > Nautilus > > > > Hello list, > > > > i upgraded to Debian 10, after that i upgraded from luminous to > > nautilus. > > I restarted the mons, then the OSDs. > > > > Everything was up and healthy. > > After rebooting a node, only 3/10 OSD start up: > > > > -4 20.07686 host ceph03 > > 4 hdd 2.67020 osd.4 down 1.00000 1.00000 > > 5 hdd 1.71660 osd.5 up 1.00000 1.00000 > > 6 hdd 1.71660 osd.6 up 1.00000 1.00000 > > 10 hdd 2.67029 osd.10 up 1.00000 1.00000 > > 15 hdd 2.00000 osd.15 down 1.00000 1.00000 > > 17 hdd 1.20000 osd.17 down 1.00000 1.00000 > > 20 hdd 1.71649 osd.20 down 1.00000 1.00000 > > 21 hdd 2.00000 osd.21 down 1.00000 1.00000 > > 27 hdd 1.71649 osd.27 down 1.00000 1.00000 > > 32 hdd 2.67020 osd.32 down 1.00000 1.00000 > > > > root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser > > ceph --setgroup ceph > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > > > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) > > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > > > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) > > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > > > failed to fetch mon config (--no-mon-config to skip) > > > > root@ceph03:~# df > > Filesystem 1K-blocks Used Available Use% Mounted on > > udev 24624580 0 24624580 0% /dev > > tmpfs 4928216 9544 4918672 1% /run > > /dev/sda3 47930248 5209760 40262684 12% / > > tmpfs 24641068 0 24641068 0% /dev/shm > > tmpfs 5120 0 5120 0% /run/lock > > tmpfs 24641068 0 24641068 0% /sys/fs/cgroup > > /dev/sda1 944120 144752 734192 17% /boot > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-1 > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-6 > > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-5 > > tmpfs 24641068 24 24641044 1% > > /var/lib/ceph/osd/ceph-10 > > tmpfs 4928212 0 4928212 0% /run/user/0 > > > > root@ceph03:~# ceph-volume lvm list > > > > > > ====== osd.1 ======= > > > > [block] > > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > > -4 > > 7bd-8ddc-102b98444067 > > > > block device > > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > > -4 > > 7bd-8ddc-102b98444067 > > block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 > > osd id 1 > > type block > > vdo 0 > > devices /dev/sdj > > > > ====== osd.10 ====== > > > > [block] > > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > > -4 > > b85-9799-988eee3b2b3f > > > > block device > > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > > -4 > > b85-9799-988eee3b2b3f > > block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid fa241441-1758-4b85-9799-988eee3b2b3f > > osd id 10 > > type block > > vdo 0 > > devices /dev/sdl > > > > ====== osd.5 ======= > > > > [block] > > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > > -4 > > e50-9bb5-775bacd854af > > > > block device > > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > > -4 > > e50-9bb5-775bacd854af > > block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af > > osd id 5 > > type block > > vdo 0 > > devices /dev/sdb > > > > ====== osd.6 ======= > > > > [block] > > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > > -4 > > 2ab-aeb0-f2915a389e48 > > > > block device > > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > > -4 > > 2ab-aeb0-f2915a389e48 > > block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM > > cephx lockbox secret > > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > > cluster name ceph > > crush device class None > > encrypted 0 > > osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 > > osd id 6 > > type block > > vdo 0 > > devices /dev/sdc > > > > I would mount it and run the osd deamon manually, but ceph-disk list > > seems to be gone in Nautilus. Therefore i dont know where to mount > what. > > > > Any ideas on that? > > > > > > Cheers, > > Michael > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > > > >

2 2

Re: OSDs wont mount on Debian 10 (Buster) with Nautilus
by Marc Roos 25 Mar '20

25 Mar '20

Try this chown ceph.ceph /dev/sdc2 chown ceph.ceph /dev/sdd2 chown ceph.ceph /dev/sde2 chown ceph.ceph /dev/sdf2 chown ceph.ceph /dev/sdg2 chown ceph.ceph /dev/sdh2 -----Original Message----- From: Ml Ml [mailto:mliebherr99@googlemail.com] Sent: 25 March 2020 16:22 To: Marc Roos Subject: Re: [ceph-users] OSDs wont mount on Debian 10 (Buster) with Nautilus They where indeed disabled. I enabled them with: systemctl enable ceph-osd@4 systemctl enable ceph-osd@15 systemctl enable ceph-osd@17 systemctl enable ceph-osd@20 systemctl enable ceph-osd@21 systemctl enable ceph-osd@27 systemctl enable ceph-osd@32 But they still wont start. On Wed, Mar 25, 2020 at 4:09 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I had something similar. My osd were disabled, maybe this installer of > nautilus does that check > > systemctl is-enabled ceph-osd@0 > > https://tracker.ceph.com/issues/44102 > > > > > -----Original Message----- > From: Ml Ml [mailto:mliebherr99@googlemail.com] > Sent: 25 March 2020 16:05 > To: ceph-users > Subject: [ceph-users] OSDs wont mount on Debian 10 (Buster) with > Nautilus > > Hello list, > > i upgraded to Debian 10, after that i upgraded from luminous to > nautilus. > I restarted the mons, then the OSDs. > > Everything was up and healthy. > After rebooting a node, only 3/10 OSD start up: > > -4 20.07686 host ceph03 > 4 hdd 2.67020 osd.4 down 1.00000 1.00000 > 5 hdd 1.71660 osd.5 up 1.00000 1.00000 > 6 hdd 1.71660 osd.6 up 1.00000 1.00000 > 10 hdd 2.67029 osd.10 up 1.00000 1.00000 > 15 hdd 2.00000 osd.15 down 1.00000 1.00000 > 17 hdd 1.20000 osd.17 down 1.00000 1.00000 > 20 hdd 1.71649 osd.20 down 1.00000 1.00000 > 21 hdd 2.00000 osd.21 down 1.00000 1.00000 > 27 hdd 1.71649 osd.27 down 1.00000 1.00000 > 32 hdd 2.67020 osd.32 down 1.00000 1.00000 > > root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser > ceph --setgroup ceph > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory > 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) > no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > root@ceph03:~# df > Filesystem 1K-blocks Used Available Use% Mounted on > udev 24624580 0 24624580 0% /dev > tmpfs 4928216 9544 4918672 1% /run > /dev/sda3 47930248 5209760 40262684 12% / > tmpfs 24641068 0 24641068 0% /dev/shm > tmpfs 5120 0 5120 0% /run/lock > tmpfs 24641068 0 24641068 0% /sys/fs/cgroup > /dev/sda1 944120 144752 734192 17% /boot > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-1 > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-6 > tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-5 > tmpfs 24641068 24 24641044 1% > /var/lib/ceph/osd/ceph-10 > tmpfs 4928212 0 4928212 0% /run/user/0 > > root@ceph03:~# ceph-volume lvm list > > > ====== osd.1 ======= > > [block] > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > -4 > 7bd-8ddc-102b98444067 > > block device > /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5 > -4 > 7bd-8ddc-102b98444067 > block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 > osd id 1 > type block > vdo 0 > devices /dev/sdj > > ====== osd.10 ====== > > [block] > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > -4 > b85-9799-988eee3b2b3f > > block device > /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758 > -4 > b85-9799-988eee3b2b3f > block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid fa241441-1758-4b85-9799-988eee3b2b3f > osd id 10 > type block > vdo 0 > devices /dev/sdl > > ====== osd.5 ======= > > [block] > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > -4 > e50-9bb5-775bacd854af > > block device > /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b > -4 > e50-9bb5-775bacd854af > block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af > osd id 5 > type block > vdo 0 > devices /dev/sdb > > ====== osd.6 ======= > > [block] > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > -4 > 2ab-aeb0-f2915a389e48 > > block device > /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6 > -4 > 2ab-aeb0-f2915a389e48 > block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM > cephx lockbox secret > cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df > cluster name ceph > crush device class None > encrypted 0 > osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 > osd id 6 > type block > vdo 0 > devices /dev/sdc > > I would mount it and run the osd deamon manually, but ceph-disk list > seems to be gone in Nautilus. Therefore i dont know where to mount what. > > Any ideas on that? > > > Cheers, > Michael > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

OSDs wont mount on Debian 10 (Buster) with Nautilus
by Ml Ml 25 Mar '20

25 Mar '20

Hello list, i upgraded to Debian 10, after that i upgraded from luminous to nautilus. I restarted the mons, then the OSDs. Everything was up and healthy. After rebooting a node, only 3/10 OSD start up: -4 20.07686 host ceph03 4 hdd 2.67020 osd.4 down 1.00000 1.00000 5 hdd 1.71660 osd.5 up 1.00000 1.00000 6 hdd 1.71660 osd.6 up 1.00000 1.00000 10 hdd 2.67029 osd.10 up 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000 root@ceph03:~# /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x56531c50a140) no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx 2020-03-25 15:46:36.330 7efddde5ec80 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-32/keyring: (2) No such file or directory 2020-03-25 15:46:36.330 7efddde5ec80 -1 AuthRegistry(0x7ffd04120468) no keyring found at /var/lib/ceph/osd/ceph-32/keyring, disabling cephx failed to fetch mon config (--no-mon-config to skip) root@ceph03:~# df Filesystem 1K-blocks Used Available Use% Mounted on udev 24624580 0 24624580 0% /dev tmpfs 4928216 9544 4918672 1% /run /dev/sda3 47930248 5209760 40262684 12% / tmpfs 24641068 0 24641068 0% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 24641068 0 24641068 0% /sys/fs/cgroup /dev/sda1 944120 144752 734192 17% /boot tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-1 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-6 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-5 tmpfs 24641068 24 24641044 1% /var/lib/ceph/osd/ceph-10 tmpfs 4928212 0 4928212 0% /run/user/0 root@ceph03:~# ceph-volume lvm list ====== osd.1 ======= [block] /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5-47bd-8ddc-102b98444067 block device /dev/ceph-9af8fc69-cab8-4c12-b51e-5746a0f0fc51/osd-block-b4987093-4fa5-47bd-8ddc-102b98444067 block uuid HSK6Da-elP2-CFYz-s0RH-UNiw-bey0-dVcml1 cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid b4987093-4fa5-47bd-8ddc-102b98444067 osd id 1 type block vdo 0 devices /dev/sdj ====== osd.10 ====== [block] /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758-4b85-9799-988eee3b2b3f block device /dev/ceph-78f2730d-7277-4d1f-8909-449b45339f80/osd-block-fa241441-1758-4b85-9799-988eee3b2b3f block uuid 440fNG-guO2-l1WJ-m5cR-GUkz-ZTUd-Fcz5Ml cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid fa241441-1758-4b85-9799-988eee3b2b3f osd id 10 type block vdo 0 devices /dev/sdl ====== osd.5 ======= [block] /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b-4e50-9bb5-775bacd854af block device /dev/ceph-793608ca-9dd1-4a4f-a776-c1e292127899/osd-block-112e0c75-f61b-4e50-9bb5-775bacd854af block uuid Z6VeNx-S9sg-ZOsh-HTw9-ykTc-YBrh-qFwz5i cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid 112e0c75-f61b-4e50-9bb5-775bacd854af osd id 5 type block vdo 0 devices /dev/sdb ====== osd.6 ======= [block] /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6-42ab-aeb0-f2915a389e48 block device /dev/ceph-4b0cee89-03f4-4853-bc1d-09e0eb772799/osd-block-35288829-c1f6-42ab-aeb0-f2915a389e48 block uuid G9qHxC-dN0b-XBes-QVss-Bzwa-7Xtw-ikksgM cephx lockbox secret cluster fsid 5436dd5d-83d4-4dc8-a93b-60ab5db145df cluster name ceph crush device class None encrypted 0 osd fsid 35288829-c1f6-42ab-aeb0-f2915a389e48 osd id 6 type block vdo 0 devices /dev/sdc I would mount it and run the osd deamon manually, but ceph-disk list seems to be gone in Nautilus. Therefore i dont know where to mount what. Any ideas on that? Cheers, Michael

2 1

March Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 25 Mar '20

25 Mar '20

Hello, We will be having a Ceph science/research/big cluster call on Wednesday March 25th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200325 Due to the worldwide effects of covid-19 I'm thinking it won't hurt to try to host this call. If only a few people join the call we can then decide to continue or cancel it. Ceph calendar event details: March 25, 2020 9am US Central 4pm Central Eurpean We try to keep it to an hour or less. Description:Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index <https://www.google.com/url?q=https://pad.ceph.com/p/Ceph_Science_User_Group…> Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink <https://www.google.com/url?q=https://bluejeans.com/908675367?src%3Dcalendar…> To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc <https://www.google.com/url?q=http://bjn.vc&sa=D&ust=1579363980705000&usg=AO…> 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers <https://www.google.com/url?q=https://www.redhat.com/en/conference-numbers&s…> 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 <https://www.google.com/url?q=https://bluejeans.com/111&sa=D&ust=15793639807…> Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 1

Re: How can I recover PGs in state 'unknown', where OSD location seems to be lost?
by Mark S. Holliman 25 Mar '20

25 Mar '20

So I've managed to use ceph-objectstore-tool to locate the pgs in 'unknown' state on the OSDs, but how do I tell the rest of the system where to find them? Is there a command for setting a the OSDs associated with a PG? Or, less ideally, is there a table somewhere I can hack to do this by hand? Mark Holliman Wide Field Astronomy Unit Institute for Astronomy University of Edinburgh The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: Ceph fully crash and we unable to recovery
by Parker Lau 25 Mar '20

25 Mar '20

Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. From: Parker Lau Sent: Wednesday, March 25, 2020 1:00 AM To: ceph-users(a)ceph.io Subject: 回覆: Ceph fully crash and we unable to recovery Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. CONFIDENTIALITY: This message is intended for the recipient named above. It may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately by replying to this message and then delete it from your system. Do not read, copy, use or circulate this communication.   SECURITY: Internet communications are not secure. While reasonable effort has been made to ensure that this communication has not been tampered with, the sender cannot be responsible for alterations made to the contents of this message without his/her express consent. The sender also cannot be responsible for any computer virus/worm/ trojan horse or other program(s) accompanying this message which he/she is unaware of. Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. 寄件者: Parker Lau 寄件日期: Wednesday, 25 March 2020 12:55 am 收件者: 'ceph-fs(a)ceph.io' <ceph-fs(a)ceph.io<mailto:ceph-fs@ceph.io>> 主旨: Ceph fully crash and we unable to recovery Hello Sir/Madam, We are facing the serious problem for our proxmox with Ceph. I have already submitted the ticket to Proxmox but they said that only option trying to recover the mondb.We would like to know any some suggestion in our situation. So far the only option that I see would be in trying to recover the mondb from an OSD. But this action is usually a last resort. Since I don't know the outcome, the cluster could then very well be dead and thus all data lost. https://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-m… --- > > I would like to set the nodown on the cluster to see if the OSDs are kept in the cluster. > > The OSDs are joining the cluster but are set as down shortly after. > > > ok . Please go ahead. Sadly this didn't have any effect either. But I think I found a clue to what might be going on. # ceph-osd.0.log 2020-03-24 21:22:06.462100 7fb33aab0e00 10 osd.0 0 read_superblock sb(e8e81549-91e5-4370-b091-9500f406a2b2 osd.0 0bb2b9bb-9a70-4d6f-8d4e-3fc5049d63d6 e14334 [13578,14334] lci=[0,14334]) # ceph-mon.cccs01.log 2020-03-24 21:26:48.038345 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 e14299: 48 total, 13 up, 35 in 2020-03-24 21:26:48.038351 7f7ef791a700 5 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 can_mark_out current in_ratio 0.729167 < min 0.75, will not mark osds out 2020-03-24 21:26:48.038360 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 tick NOOUT flag set, not checking down osds 2020-03-24 21:26:48.038364 7f7ef791a700 10 mon.cccs01(a)0(leader).osd<mailto:mon.cccs01@0(leader).osd> e14299 min_last_epoch_clean 0 # ceph-mon.cccs06.log 2020-03-22 22:26:57.056939 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14333 e14333: 48 total, 48 up, 48 in 2020-03-22 22:27:04.113054 7f3c1993a700 0 mon.cccs06@5(peon)<mailto:mon.cccs06@5(peon)> e31 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1 2020-03-22 22:27:04.113086 7f3c1993a700 0 log_channel(audit) log [DBG] : from='client.? 10.1.14.8:0/4265796352' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-03-22 22:27:09.752027 7f3c1993a700 1 mon.cccs06(a)5(peon).osd<mailto:mon.cccs06@5(peon).osd> e14334 e14334: 48 total, 48 up, 48 in ... 2020-03-23 10:42:51.891722 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891729 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires 2020-03-23 10:42:51.891730 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 1009089991638532096, adjusting msgr requires 2020-03-23 10:42:51.891732 7ff1d9079700 0 mon.cccs06(a)2(synchronizing).osd<mailto:mon.cccs06@2(synchronizing).osd> e14269 crush map has features 288514051259236352, adjusting msgr requires It seems that the OSD have a epoch of e14334 but the MONs seems to have e14269 for the OSDs. I could only find the e14334 on the ceph-mon.cccs06. The cccs06 has been the last MON standing (the node also reset). But when the cluster came back the MONs with the old epoch came up first and joined. The syslog showed that these were the last log entries written. So the nodes reset shortly after. This would fit to cccs06 being the last MON alive. # cccs01 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: leader is 1/2502 Mar 22 22:16:09 cccs01 pmxcfs[2502]: [dcdb] notice: synced members: 1/2502, 5/2219 # cccs02 Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] notice: we (3/2514) left the process group Mar 22 22:15:57 cccs02 pmxcfs[2514]: [dcdb] crit: leaving CPG group # cccs06 Mar 22 22:31:16 cccs06 pmxcfs[2662]: [status] noMar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'iscsi_tcp' Mar 22 22:34:01 cccs06 systemd-modules-load[773]: Inserted module 'ib_iser' There must have been some issue prior to the reset as I found those error messages. They could explain why no older epoch was written anymore. # cccs01 2020-03-22 22:22:15.661060 7fc09c85c100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs01/store.db/LOCK: Permission denied 2020-03-22 22:22:15.661067 7fc09c85c100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs01': (22) Invalid argument # cccs02 2020-03-22 22:31:11.209524 7fd034786100 -1 rocksdb: IO error: /var/lib/ceph/mon/ceph-cccs02/store.db/LOCK: Permission denied 2020-03-22 22:31:11.209541 7fd034786100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-cccs02': (22) Invalid argument # cccs06 no such entries. From my point of view, the last question is now: How to get the epoch from the OSD into the MON DB. I have no answer to this yet. Best Regards, Parker Lau ReadySpace Ltd - Cloud and Managed Hosting Professionals Direct: +852 3726 1120 Hotiline: +852 3568 3372<tel:+852%203568%203372> Fax: +852 3568 3376<tel:+852%203568%203376> Website: www.readyspace.com.hk<http://www.readyspace.com.hk/> Helpdesk: helpdesk.readyspace.com<http://helpdesk.readyspace.com/> Get the latest update and promotion here: twitter.com/readyspace<http://twitter.com/readyspace> | facebook.com/readyspace<http://www.facebook.com/readyspace> [cid:image001.jpg@01D5CC61.CF90E050]Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately. CONFIDENTIALITY: This message is intended for the recipient named above. It may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately by replying to this message and then delete it from your system. Do not read, copy, use or circulate this communication.   SECURITY: Internet communications are not secure. While reasonable effort has been made to ensure that this communication has not been tampered with, the sender cannot be responsible for alterations made to the contents of this message without his/her express consent. The sender also cannot be responsible for any computer virus/worm/ trojan horse or other program(s) accompanying this message which he/she is unaware of. Please consider the environment before printing this email. Information in this message is confidential. It is intended solely for the person or the entity to whom it is addressed. If you are not the intended recipient, you are not to disseminate, distribute or copy this communication. Please notify the sender and delete the message and any other record of it from your system immediately.

1 0

multi-node NFS Ganesha + libcephfs caching
by Maged Mokhtar 24 Mar '20

24 Mar '20

Hello all, For multi-node NFS Ganesha over CephFS, is it OK to leave libcephfs write caching on, or should it be configured off for failover ? Cheers /Maged

3 8

rbd-mirror -> how far behind_master am i time wise?
by Ml Ml 24 Mar '20

24 Mar '20

Hello List, i use rbd-mirror and i asynchronously mirror to my backup cluster. My backup cluster only has "spinnung rust" and wont be able to always perform like the live cluster. Thats is fine for me, as far as it´s not further behind than 12h. vm-194-disk-1: global_id: 7a95730f-451c-4973-8038-2a59e29ac5ad state: up+replaying description: replaying, master_position=[object_number=1046, tag_tid=4, entry_tid=936210], mirror_position=[object_number=911, tag_tid=4, entry_tid=815131], entries_behind_master=121079 last_update: 2020-03-24 08:43:43 I learned, that the entries_behind_master are single transactions. But what i am really interested in is: How far am i behind time wise? Is there a way to tell his? Thanks, Michael

2 1

French Classes in Bangalore
by Ria institute 24 Mar '20

24 Mar '20

RIA Institute of Technology Provides easy to understand French Language Classes in Bangalore. Learning French is considered to be a part of the curriculum for many professional jobs in major Multinational companies. We offer French Training in Marathahalli, Bangalore to aspiring Professionals and Students who are looking forward to upgrade their skills in Foreign Language. We offer French Training Classes to Educational Segments such as First and Second PUC College Students, Working Professionals, Business Travellers and Language learning enthusiasts. Structured Course content and individual focused approach has enabled us to be one of the Best Training Institutes in Bangalore when it comes to French Language Courses. Our Trainers are Professionals in their field with relative experience. https://www.riainstitute.co.in/French-Language-Training-in-Bangalore.html

1 0

Fwd: RGW failing to create bucket
by Abhinav Singh 24 Mar '20

24 Mar '20

---------- Forwarded message --------- From: Abhinav Singh <singhabhinav0796(a)gmail.com> Date: Mon, Mar 23, 2020 at 7:43 PM Subject: RGW failing to create bucket To: <dev(a)ceph.io> ceph : octopus JaegerTracing : master ubuntu : 18.04 When I implementing jaeger tracing it is unable to create a bucket. (I m using swif to perform testing.) /src/librados/IoCtxImpl.cc ``` void librados::IoCtxImpl::queue_aio_write(AioCompletionImpl *c) { std::cout<<"yes"<<std::endl; JTracer tracer; tracer.initTracer("Writing Started", "/home/abhinav/Desktop/GSOC/deepika/ceph/src/librados/tracerConfig.yaml"); Span span=tracer.newSpan("writing started"); span->Finish(); try{ auto yaml = YAML::LoadFile("tracerConfig.yaml"); }catch(const YAML::ParserException& pe){ // ldout<<pe.what()<<dendl; std::cout<<pe.what()<<std::endl; ofstream f; f.open("/home/abhinav/Desktop/err.txt"); f<<pe.what(); f.close(); } // auto config = jaegertracing::Config::parse(yaml); // auto tracer=jaegertracing::Tracer::make( // "Writing", // config, // jaegertracing::logging::consoleLogger() // ); // opentracing::Tracer::InitGlobal( // static_pointer_cast<opentracing::Tracer>(tracer) // ); // auto span = opentracing::Tracer::Global()->StartSpan("Span1"); get(); ofstream file; file.open("/home/abhinav/Desktop/write.txt",std::ios::out | std::ios::app); file<<"Writing /src/librados/IoCtxImpl.cc 310.\n"; file.close(); std::scoped_lock l{aio_write_list_lock}; ceph_assert(c->io == this); c->aio_write_seq = ++aio_write_seq; ldout(client->cct, 20) << "queue_aio_write " << this << " completion " << c << " write_seq " << aio_write_seq << dendl; aio_write_list.push_back(&c->aio_write_list_item); // opentracing::Tracer::Global()->Close(); } ``` /include/tracer.h ``` typedef std::unique_ptr<opentracing::Span> Span; class JTracer{ public: JTracer(){} ~JTracer(){ opentracing::Tracer::Global()->Close(); } void static inline loadYamlConfigFile(const char* path){ return; } void initTracer(const char* tracerName,const char* filePath){ auto yaml = YAML::LoadFile(filePath); auto configuration = jaegertracing::Config::parse(yaml); auto tracer = jaegertracing::Tracer::make( tracerName, configuration, jaegertracing::logging::consoleLogger()); opentracing::Tracer::InitGlobal( std::static_pointer_cast<opentracing::Tracer>(tracer)); Span s=opentracing::Tracer::Global()->StartSpan("Testing"); s->Finish(); } Span newSpan(const char* spanName){ Span span=opentracing::Tracer::Global()->StartSpan(spanName); return std::move(span); } Span childSpan(const char* spanName,const Span& parentSpan){ Span span = opentracing::Tracer::Global()->StartSpan(spanName, {opentracing ::ChildOf(&parentSpan->context())}); return std::move(span); } Span followUpSpan(const char *spanName, const Span& parentSpan){ Span span = opentracing::Tracer::Global()->StartSpan(spanName, {opentracing ::FollowsFrom(&parentSpan->context())}); return std::move(span); } }; ``` Output when trying to create new container ``` errno 111 connection refused ``` But when I remove the tracer part in IoCtxImpl.cc it is workng fine. I m new to ceph, and dont know what information to share to correctly track down the problem, if any extra informtion is needed I will share it instantly. Been stuck into this issue for one week. Please someone help me! Thank you.

2 3

Re: Newbie to Ceph jacked up his monitor
by Jarett DeAngelis 24 Mar '20

24 Mar '20

So, I thought I’d post with what I learned re: what to do with this problem. This system is a 3-node Proxmox cluster, and each node had: 1 x 1TB NVMe 2 x 512GB HDD I had maybe 100GB of data in this system total. Then I added: 2 x 256GB SSD 1 x 1TB HDD To each system, and let it start rebalancing. When it started the management interface showed the storage as being out of order in various ways, but it was clear that Ceph was rebalancing PGs across the 3 nodes and the “broken” part of the graphic display was shrinking as it spread data across the added OSDs. In the process, however, the monitors racked up ENORMOUS amounts of files. On one machine, the boot drive only has 64GB of space total so the partition where /var/lib/ceph/somethingsomething.db lived was only 27GB. This filled up very, very fast, and eventually killed the monitor on that node. I figured out you can `ceph-monstore-tool compact` or `ceph-kvstore-tool rocksdb /path compact` to get the system to truncate the files in there, but even when I scheduled those jobs to run on each monitor every minute the amount of space being taken up by those rocksdb files grew and grew until they threatened to kill the monitors on the nodes with larger amounts of space too. Other, dumber measures I tried taking to give the system more space for these files ended up screwing up my Proxmox system, so now I have to reinstall. What can be done about this problem so that I don’t have this issue when I try to implement again? Thanks!

2 1

Re: MGRs failing once per day and generally slow response times
by Janek Bevendorff 24 Mar '20

24 Mar '20

Indeed. I just had another MGR go bye-bye. I don't think host clock skew is the problem. On 13/03/2020 15:29, Anthony D'Atri wrote: > Chrony does converge faster, but I doubt this will solve your problem if you don’t have quality peers. Or if it’s not really a time problem. > >> On Mar 13, 2020, at 6:44 AM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: >> >> I replaced ntpd with chronyd and will let you know if it changes anything. Thanks. >> >> >>> On 13/03/2020 06:25, Konstantin Shalygin wrote: >>>> On 3/13/20 12:57 AM, Janek Bevendorff wrote: >>>> NTPd is running, all the nodes have the same time to the second. I don't think that is the problem. >>> As always in such cases - try to switch your ntpd to default EL7 daemon - chronyd. >>> >>> >>> >>> k >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Bauhaus-Universität Weimar Bauhausstr. 9a, Room 308 99423 Weimar, Germany Phone: +49 (0)3643 - 58 3577

1 3

Re: Cephfs mount error 1 = Operation not permitted
by Eugen Block 24 Mar '20

24 Mar '20

> I suppose the correct syntax is that anything after "client." is the > name? So: > > ceph fs authorize cephfs client.bob / r / rw > > Would authorize a client named bob? Yes, exactly: admin:~ # ceph fs authorize cephfs client.bob / r / rw [client.bob] key = AQAyw3leAv9tKxAA+wtNEa40yK6svPE/VPlqdA== admin:~ # mount -t ceph mon1:/ /mnt/ -o name=bob,secret=AQAyw3leAv9tKxAA+wtNEa40yK6svPE/VPlqdA== admin:~ # touch /mnt/file Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > That was it! I am not sure how I got confused with the client name > syntax. When I issued the command to create a client key, I used: > > ceph fs authorize cephfs client.1 / r / rw > > I assumed from the syntax that my client name is "client.1" > > I suppose the correct syntax is that anything after "client." is the > name? So: > > ceph fs authorize cephfs client.bob / r / rw > > Would authorize a client named bob? > > -Scott > ________________________________ > From: Eugen Block <eblock(a)nde.ag> > Sent: Monday, March 23, 2020 11:30 AM > To: Dungan, Scott A. <sdungan(a)caltech.edu> > Cc: Yan, Zheng <ukernel(a)gmail.com>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted > > Wait, your client name is just "1"? In that case you need to specify > that in your mount command: > > mount ... -o name=1,secret=... > > It has to match your ceph auth settings, where "client" is only a > prefix and is followed by the client's name > > [client.1] > > > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: > >> Tried that: >> >> [client.1] >> key = ******************************* >> caps mds = "allow rw path=/" >> caps mon = "allow r" >> caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" >> >> No change. >> >> >> ________________________________ >> From: Yan, Zheng <ukernel(a)gmail.com> >> Sent: Sunday, March 22, 2020 9:28 PM >> To: Dungan, Scott A. <sdungan(a)caltech.edu> >> Cc: Eugen Block <eblock(a)nde.ag>; ceph-users(a)ceph.io <ceph-users(a)ceph.io> >> Subject: Re: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted >> >> On Sun, Mar 22, 2020 at 8:21 AM Dungan, Scott A. >> <sdungan(a)caltech.edu> wrote: >>> >>> Zitat, thanks for the tips. >>> >>> I tried appending the key directly in the mount command >>> (secret=<CLIENT.1.SECRET>) and that produced the same error. >>> >>> I took a look at the thread you suggested and I ran the commands >>> that Paul at Croit suggested even though I the ceph dashboard >>> showed "cephs" as already set as the application on both my data >>> and metadata pools: >>> >>> [root@ceph-n4 ~]# ceph osd pool application set data cephfs data cephfs >>> set application 'cephfs' key 'data' to 'cephfs' on pool 'data' >>> [root@ceph-n4 ~]# ceph osd pool application set meta_data cephfs >>> metadata cephfs >>> set application 'cephfs' key 'metadata' to 'cephfs' on pool 'meta_data' >>> >>> No change. I get the "mount error 1 = Operation not permitted" >>> error the same as before. >>> >>> I also tried manually editing the caps osd pool tags for my >>> client.1, to allow rw to both the data pool as well as the metadata >>> pool, as suggested further in the thread: >>> >>> [client.1] >>> key = *********************************** >>> caps mds = "allow rw path=all" >> >> >> try replacing this with "allow rw path=/" >> >>> caps mon = "allow r" >>> caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data" >>> >>> No change. >>> >>> ________________________________ >>> From: Eugen Block <eblock(a)nde.ag> >>> Sent: Saturday, March 21, 2020 1:16 PM >>> To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> >>> Subject: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted >>> >>> I just remembered there was a thread [1] about that a couple of weeks >>> ago. Seems like you need to add the capabilities to the client. >>> >>> [1] >>> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… >>> >>> >>> Zitat von Eugen Block <eblock(a)nde.ag>: >>> >>> > Hi, >>> > >>> > have you tried to mount with the secret only instead of a secret file? >>> > >>> > mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=<SECRET> >>> > >>> > If that works your secret file is not right. If not you should check >>> > if the client actually has access to the cephfs pools ('ceph auth >>> > list'). >>> > >>> > >>> > >>> > Zitat von "Dungan, Scott A." <sdungan(a)caltech.edu>: >>> > >>> >> I am still very new to ceph and I have just set up my first small >>> >> test cluster. I have Cephfs enabled (named cephfs) and everything >>> >> is good in the dashboard. I added an authorized user key for cephfs >>> >> with: >>> >> >>> >> ceph fs authorize cephfs client.1 / r / rw >>> >> >>> >> I then copied the key to a file with: >>> >> >>> >> ceph auth get-key client.1 > /tmp/client.1.secret >>> >> >>> >> Copied the file over to the client and then attempt mount witth the >>> >> kernel driver: >>> >> >>> >> mount -t ceph ceph-n4:6789:/ /ceph -o >>> >> name=client.1,secretfile=/root/client.1.secret >>> >> mount error 1 = Operation not permitted >>> >> >>> >> I looked in the logs on the mds (which is also the mgr and mon for >>> >> the cluster) and I don't see any events logged for this. I also >>> >> tried the mount command with verbose and I didn't get any further >>> >> detail. Any tips would be most appreciated. >>> >> >>> >> -- >>> >> >>> >> Scott Dungan >>> >> California Institute of Technology >>> >> Office: (626) 395-3170 >>> >> sdungan(a)caltech.edu<mailto:sdungan@caltech.edu> >>> >> >>> >> _______________________________________________ >>> >> ceph-users mailing list -- ceph-users(a)ceph.io >>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Building a Ceph cluster with Ubuntu 18.04 and NVMe SSDs
by Georg Schönberger 24 Mar '20

24 Mar '20

Hi Ceph users! We are currently configuring our new production Ceph cluster and I have some questions regarding Ubuntu and NVMe SSDs. Basic setup: - Ubuntu 18.04 with HWE Kernel 5.3 - Deployment via ceph-ansible (Ceph stable "Nautilus") - 5x Nodes with AMD EPYC 7402P CPUs - 25Gbit/s NICs and switches for Ceph private and public network - 4x Intel P4510 2TB NVMe SSDs (all flash) per Node My questions: 1. Should we deploy more than one OSD per NVMe SSD? (as P4510's performance can sustain e.g. 2 OSDs) 2. Does anyone know NVMe specific Linux settings we should enable? 3. Can we use io_uring, if yes how can we enable it? Is it enough to set bluestore_iouring=true? What I know so far: Ad 1: My opinion is to use at least 2 OSDs per NVMEe SSD as the Intel P4510 is fast enough to serve the parallel requests. Please be aware to use the latest firmware version VDV10170 -> with version VDV10131 we had massive stalls on Ceph side! Ad 2: I have already enabled NVMe polling queues, Ubuntu has disabled them by default: Added nvme.poll_queues=1 to /etc/default/grub, then checked /sys/block/nvme1n1/queue/io_poll Cf. https://lore.kernel.org/linux-block/20190318222133.GA24176@localhost.locald… Ad 3: This commit states it should be possible to use io_uring: https://github.com/ceph/ceph/pull/27392 This issue also shows how to set bluestore_iouring=true but it's not clear if any more setup is required, like liburing: https://github.com/axboe/liburing A presentation from Christoph Hellwig shows the advantages: https://www.snia.org/sites/default/files/SDC/2019/presentations/NVMe/Hellwi… Any help and inputs would be appreciated, THX - Georg

1 0

Exporting
by Rhian Resnick 24 Mar '20

24 Mar '20

Evening, We are running into issues exporting a disk image from ceph rbd. When we attempt to export an rbd image in a cache tiered erasure-coded pool on Luminus. All the other disks are working fine but this one is acting up. We have a bit of important data on other disks so obviously want to make sure this doesn't happen to those. [root@ceph-p-mon1 home]# rbd export one/one-177-588-0 one-177-588-0 Exporting image: 8% complete...rbd: error reading from source image at offset 5456789504: (5) Input/output error 2020-03-23 20:11:29.210718 7f2f3effd700 -1 librbd::io::ObjectRequest: 0x7f2f2c128f90 handle_read_object: failed to read from object: (5) Input/output error 2020-03-23 20:11:29.565184 7f2f3e7fc700 -1 librbd::io::ObjectRequest: 0x7f2f280c84d0 handle_read_cache: failed to read from cache: (5) Input/output error Exporting image: 8% complete...failed. rbd: export error: (5) Input/output error Any thoughts would be appreciated. Some info: [root@ceph-p-mon1 home]# rbd info one/one-177-588-0 rbd image 'one-177-588-0': size 58.6GiB in 15000 objects order 22 (4MiB objects) block_name_prefix: rbd_data.84a01279e2a9e3 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: create_timestamp: Fri Apr 20 17:06:09 2018 parent: one/one-177@snap overlap: 2.20GiB [root@ceph-p-mon1 home]# ceph status cluster: id: 6a2e8f21-bca2-492b-8869-eecc995216cc health: HEALTH_OK services: mon: 3 daemons, quorum ceph-p-mon2,ceph-p-mon1,ceph-p-mon3 mgr: ceph-p-mon2(active) mds: cephfsec-1/1/1 up {0=ceph-p-mon2=up:active}, 6 up:standby osd: 155 osds: 154 up, 154 in data: pools: 6 pools, 5904 pgs objects: 145.53M objects, 192TiB usage: 253TiB used, 290TiB / 543TiB avail pgs: 5896 active+clean 8 active+clean+scrubbing+deep io: client: 921KiB/s rd, 5.68MiB/s wr, 110op/s rd, 29op/s wr cache: 5.65MiB/s flush, 0op/s promote [root@ceph-p-mon1 home]# rpm -qa | grep ceph ceph-common-12.2.9-0.el7.x86_64 ceph-mds-12.2.9-0.el7.x86_64 ceph-radosgw-12.2.9-0.el7.x86_64 ceph-mgr-12.2.9-0.el7.x86_64 ceph-12.2.9-0.el7.x86_64 collectd-ceph-5.8.1-1.el7.x86_64 ceph-deploy-2.0.1-0.noarch libcephfs2-12.2.9-0.el7.x86_64 python-cephfs-12.2.9-0.el7.x86_64 ceph-selinux-12.2.9-0.el7.x86_64 ceph-osd-12.2.9-0.el7.x86_64 ceph-base-12.2.9-0.el7.x86_64 ceph-mon-12.2.9-0.el7.x86_64 ceph-release-1-1.el7.noarch Rhian Resnick Associate Director Research Computing Enterprise Systems Office of Information Technology Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg>

1 0