June 2020 - ceph-users - lists.ceph.io

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

v15.2.4 Octopus released
by David Galloway 17 Dec '20

17 Dec '20

We're happy to announce the fourth bugfix release in the Octopus series. In addition to a security fix in RGW, this release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-4-octopus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * Cephadm: There were a lot of small usability improvements and bug fixes: * Grafana when deployed by Cephadm now binds to all network interfaces. * `cephadm check-host` now prints all detected problems at once. * Cephadm now calls `ceph dashboard set-grafana-api-ssl-verify false` when generating an SSL certificate for Grafana. * The Alertmanager is now correctly pointed to the Ceph Dashboard * `cephadm adopt` now supports adopting an Alertmanager * `ceph orch ps` now supports filtering by service name * `ceph orch host ls` now marks hosts as offline, if they are not accessible. * Cephadm can now deploy NFS Ganesha services. For example, to deploy NFS with a service id of mynfs, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:: ceph orch apply nfs mynfs nfs-ganesha nfs-ns * Cephadm: `ceph orch ls --export` now returns all service specifications in yaml representation that is consumable by `ceph orch apply`. In addition, the commands `orch ps` and `orch ls` now support `--format yaml` and `--format json-pretty`. * Cephadm: `ceph orch apply osd` supports a `--preview` flag that prints a preview of the OSD specification before deploying OSDs. This makes it possible to verify that the specification is correct, before applying it. * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, and `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * RBD: The name of the rbd pool object that is used to store rbd trash purge schedule is changed from "rbd_trash_trash_purge_schedule" to "rbd_trash_purge_schedule". Users that have already started using `rbd trash purge schedule` functionality and have per pool or namespace schedules configured should copy "rbd_trash_trash_purge_schedule" object to "rbd_trash_purge_schedule" before the upgrade and remove "rbd_trash_purge_schedule" using the following commands in every RBD pool and namespace where a trash purge schedule was previously configured:: rados -p <pool-name> [-N namespace] cp rbd_trash_trash_purge_schedule rbd_trash_purge_schedule rados -p <pool-name> [-N namespace] rm rbd_trash_trash_purge_schedule or use any other convenient way to restore the schedule after the upgrade. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7447c15c6ff58d7fce91843b705a268a1917325c -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

4 5

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early
by Wido den Hollander 28 Sep '20

28 Sep '20

Hi, On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing OSDs randomly drop out of the cluster. Usually it's 2 to 4 OSDs spread out over different nodes. Each node has 16 OSDs and not all the failing OSDs are on the same node. The OSDs are marked as down and all they keep print in their logs: monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-06-04T07:57:17.706529-0400) Looking at their status through the admin socket: { "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", "whoami": 206, "state": "active", "oldest_map": 73697, "newest_map": 75795, "num_pgs": 19 } The message brought me to my own ticket I created 2 years ago: https://tracker.ceph.com/issues/23460 The first thing I've checked is NTP/time. Double, triple check this. All the times are in sync on the cluster. Nothing wrong there. Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. Restarting them brings them back right away and then within 24h some other OSDs will drop out. Has anybody seen this behavior with Octopus as well? Wido

2 1

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

cephadm - How to deploy ceph cluster with a partition on SSD for block.db
by klemen＠psi-net.si 11 Sep '20

11 Sep '20

I'm trying to deploy a ceph cluster with a cephadm tool. I've already successfully done all steps except adding OSDs. My testing equipment consists of three hosts. Each host has SSD storage, where OS is installed into. On that storage I created partition, which can be used as a ceph block.db. Hosts have also 2 additional HDs (spinning drives) for OSD data. On docs I couldn't find how to deploy such configuration. Do you have any hints, how to do that? Thanks for help!

5 7

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay)
by Jake Grimmett 27 Aug '20

27 Aug '20

Dear all, After enabling "allow_standby_replay" on our cluster we are getting (lots) of identical errors on the client /var/log/messages like Apr 29 14:21:26 hal kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay) We are using the ml kernel 5.6.4-1.el7 on Scientific Linux 7.8 Cluster and client are running Ceph v14.2.9 Setting was enabled with: # ceph fs set cephfs allow_standby_replay true [root@ceph-s1 ~]# ceph mds stat cephfs:1 {0=ceph-s3=up:active} 1 up:standby-replay 2 up:standby Is this something to worry about, or should we just disable allow_standby_replay ? any advice appreciated, many thanks Jake Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

2 2

OSDs get full with bluestore logs
by Khodayar Doustar 21 Aug '20

21 Aug '20

Hi, I have a 3 node cluster of mimic with 9 osds (3 osds on each node). I use this cluster to test integration of an application with S3 api. The problem is that after a few days all OSD starts filling up with bluestore logs and goes down and out one by one! I cannot stop the logs and I cannot find the setting to fix this leakage, this should be a leakage in logs because it's not logical to fill up all OSD with bluefs logs. This is an example of logs which is being repeated in bluestore logs: [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path /var/lib/ceph/osd/ceph-5 . . . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 15:54:29.056488 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 15:54:30.057828 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 15:54:31.059252 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0xf5f0) [0x7f40fd47f5f0] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] 4: (BlueFS::log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x3b) [0x55603a1fad3b] 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x64) [0x55603a1da764] 6: (main()+0x2f3e) [0x55603a105e3e] 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] 8: (()+0x23115f) [0x55603a1d915f] 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- . . . And this is the output of daemonperf for one of remaining (filling up!) OSDs: [root@server2 ~]# ceph daemonperf osd.5 ------bluefs------- ------------bluestore------------- ----------osd----------- jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l rop | 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 ^C[root@server2 ~]# And this is the enospc logs when trying to start OSD: [root@server2 ~]# journalctl -u -f ceph-osd(a)4.service Failed to add match 'ceph-osd(a)4.service': Invalid argument Failed to add filters: Invalid argument [root@server2 ~]# journalctl -f -u ceph-osd(a)4.service -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10] Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon osd.4. Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon osd.4. and some more data: [root@server1 ~]# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 38 GiB 35 GiB 2.6 GiB 6.83 303 POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED .rgw.root 18 N/A N/A 1.1 KiB 0 50 GiB 4 4 63 B 4 B 1.1 KiB default.rgw.meta 19 N/A N/A 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182 B 9.0 KiB default.rgw.log 20 N/A N/A 0 B 0 50 GiB 107 107 307 KiB 205 KiB 0 B default.rgw.control 21 N/A N/A 0 B 0 50 GiB 5 5 0 B 0 B 0 B default.rgw.buckets.index 22 N/A N/A 0 B 0 50 GiB 5 5 501 B 60 B 0 B default.rgw.buckets.data 23 N/A N/A 377 MiB 0.39 50 GiB 160 160 385 B 1.1 KiB 603 MiB [root@server1 ~]# [root@server1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server1 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.0 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.1 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.2 -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 0 - host server2 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.3 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.4 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 1.04 99 osd.5 -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server3 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 0.96 78 osd.6 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.7 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.8 TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 MIN/MAX VAR: 0/1.04 STDDEV: 5.58 [root@server1 ~]# I'm kind of newbie to ceph, so any help or hint would be appreciated. Did I hit a bug or something is wrong with my configuration? Thanks a lot, Khodayar

5 5

Help
by Randy Morgan 17 Aug '20

17 Aug '20

We are seeking information on configuring Ceph to work with Noobaa and NextCloud. Randy -- Randy Morgan CSR Department of Chemistry/BioChemistry Brigham Young University randym(a)chem.byu.edu

3 2

Ceph Nautilus packages for Ubuntu Focal
by Stefan Kooman 17 Aug '20

17 Aug '20

Hi list, We're wondering if Ceph Nautilus packages will be provided for Ubuntu Focal Fossa (20.04)? You might wonder why one would not just use Ubuntu Bionic (18.04) instead of using the latest LTS. Here is why: a glibc bug in Ubuntu Bionic that *might* affect Open vSwitch (OVS) users [1]. We had quite a few issues with OVS deadlocks on hypervisors, and do not want to risk experiencing the same issues on our Ceph cluster(s). I'm not sure how many of you use OVS for bridging / bonding, but for those who do, running Ceph (Nautlilus / Octopus) on 20.04 would be preferred. Gr. Stefan [1]: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592

2 1

Ceph influxDB support versus Telegraf Ceph plugin?
by victorhooi＠yahoo.com 06 Aug '20

06 Aug '20

Hi, I've read that Ceph has some InfluxDB reporting capabilities inbuilt (https://docs.ceph.com/docs/master/mgr/influx/) However, Telegraf, which is the system reporting daemon for InfluxDB, also has a Ceph plugin (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph) Just curious what people's thoughts on the two are, or what they are using in production? Which is easier to deploy/maintain, have you found? Or more useful for alerting, or tracking performance gremlins? Thanks, Victor

2 2

Push config to all hosts
by Cem Zafer 27 Jul '20

27 Jul '20

Hi, What is the best method(s) to push ceph.conf to all hosts in octopus (15.x)? Thanks.

2 1

IPv6 connectivity gone for Ceph Telemetry
by Wido den Hollander 24 Jul '20

24 Jul '20

Hi, I was just checking on a few (13) IPv6-only Ceph clusters and I noticed that they couldn't send their Telemetry data anymore: telemetry.ceph.com has address 8.43.84.137 This server used to have Dual-Stack connectivity while it was still hosted at OVH. It seemed to have moved to Red Hat, but lost IPv6 connectivity. How can we get this back? Wido

2 1

Problem with OSD::osd_op_tp thread had timed out and other connected issues
by Jan Pekař - Imatic 21 Jul '20

21 Jul '20

Hello, I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network Cluster was empty, fresh install. We filled cluster with data (small blocks) using RGW. Cluster is now used for testing so no client was using it during my admin operations mentioned below After a while (7TB of data / 40M objects uploaded) we decided, that we increase pg_num from 128 to 256 to better spread data and to speedup this operation, I've set ceph config set mgr target_max_misplaced_ratio 1 so that whole cluster rebalance as quickly as it can. I have 3 issues/questions below: 1) I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to restart with logged heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide timed out after 150 after a while OSD's were back so I continued after a while with my tests. My question - increasing number of PG with maximal target_max_misplaced_ratio was too much for that OSDs? It is not recommended to do it this way? I had no problem with this increase before, but configuration of cluster was slightly different and it was luminous version. 2) Rebuild was still slow so I increased number of backfills ceph tell osd.* injectargs "--osd-max-backfills 10" and reduced recovery sleep time ceph tell osd.* injectargs "--osd-recovery-sleep-hdd 0.01" and after few hours I noticed, that some of my OSD's were restarted during recovery, in log I can see ... |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 start_waiting_for_healthy | I observed network graph usage and network utilization was low during recovery (10Gbit was not saturated). So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that OSD is using threads and HDD timeouts are not influencing heartbeats to other OSD's and MON. It looks like it is not true. 3) After OSD was wrongly marked down I can see that cluster has object degraded. There were no degraded object before that. Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs degraded, 8 pgs undersized It means that this OSD disconnection causes data degraded? How is it possible, when no OSD was lost. Data should be on that OSD and after peering should be everything OK. With luminous I had no problem, after OSD up degraded objects where recovered/found during few seconds and cluster was healthy within seconds. Thank you very much for additional info. I can perform additional tests you recommend because cluster is used for testing purpose now. With regards Jan Pekar -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz | +420326555326 ============ --

4 7

bluestore_default_buffered_write = true
by Adam Koczarski 20 Jul '20

20 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

Degradation of write-performance after upgrading to Octopus
by Thomas Gradisnik 20 Jul '20

20 Jul '20

We have deployed a small test cluster consisting of three nodes. Each node is running a mon/mgr and two osds (Samsung PM983 3,84TB NVMe split into two partitions), so six osds in total. We started with Ceph 14.2.7 some weeks ago (upgraded to 14.2.9 later) and ran different tests using fio against some rbd volumes in order to get an overview what performance we could expect. The configuration is unchanged compared to the defaults, we only set several debugging options to 0/0. Yesterday we upgraded the whole cluster following the upgrade guidelines to Ceph 15.2.3, which worked without any problems so far. Nevertheless when running the same tests as before with Ceph 14.2.9, we are seeing some clear degradations in write-performance (beside some performance improvements, which shall also be mentioned). Here the results of concern (each with the relevant fio settings used): Test "read-latency-max" (rw=randread, iodepth=64, bs=4k) read_iops: 32500 -> 87000 Test "write-latency-max" (rw=randwrite, iodepth=64, bs=4k) write_iops: 22500 -> 11500 Test "write-throughput-iops-max" (rw=write, iodepth=64, bs=4k) write_iops: 7000 -> 14000 Test "usecase1" (rw=randrw, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/,4k/50:8k/20:16k/20:32k/5:64k/2:128k/:256k/, rwmixread=1, rate_process=poisson, iodepth=64) write_iops: 21000 -> 8500 Test "usecase1-readonly" (rw=randread, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/, rate_process=poisson, iodepth=64) read_iops: 28000 -> 58000 The last two tests represent a typical use case on our systems. Therefore we are especially concerned by the drop in performance from 21000 w/ops to 8500 w/ops (about 60%) after upgrading to Ceph 15.2.3. We ran all tests several times, the values are averaged over all iterations and fairly consistent and reproducible. We even tried wiping the whole cluster, downgrading to Ceph 14.2.9 again, setting up a new cluster/pool, running the tests and upgrading to Ceph 15.2.3 again. The tests have been performed on one of the three cluster nodes using a 50G rbd volume, which had been prefilled with random data before each test-run. Have any changes been introduced with Octopus that could explain the observed changes in performance? What we already tried: - Disabling rbd cache - Reverting rbc cache policy to writeback (default in 14.2) - Setting rbd io scheduler to none - Deploying a fresh cluster starting with Ceph 15.2.3 Kernel is 5.4.38 … I don't know if some other system specs would be helpful besides the already mentioned (since we are talking about a relative change in performance after upgrading Ceph without any further changes) - if so, please let us know.

6 10

Poor Windows performance on ceph RBD.
by Frank Schilder 13 Jul '20

13 Jul '20

Dear all, maybe someone can give me a pointer here. We are running OpenNebula with ceph RBD as a back-end store. We have a pool of spinning disks to create large low-demand data disks, mainly for backups and other cold storage. Everything is fine when using linux VMs. However, Windows VMs perform poorly, they are like a factor 20 slower than a similarly created linux VM. If anyone has pointers what to look for, we would be very grateful. The OpenNebula installation is more or less default. The current OS and libvirt versions we use are: Centos 7.6 with stock kernel 3.10.0-1062.1.1.el7.x86_64 libvirt-client.x86_64 4.5.0-23.el7_7.1 @updates qemu-kvm-ev.x86_64 10:2.12.0-33.1.el7 @centos-qemu-ev Some benchmark results from good to worse workloads: rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 16 : 450MB/s rbd bench --io-size 4M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 230MB/s rbd bench --io-size 1M --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 190MB/s rbd bench --io-size 64K --io-total 4G --io-pattern seq --io-type write --io-threads 1 : 150MB/s rbd bench --io-size 64K --io-total 1G --io-pattern rand --io-type write --io-threads 1 : 26MB/s dd with conv=fdatasync gives awesome 500MB/s inside linux VM for sequential write of 4GB. We copied a couple of large ISO files inside the Windows VM and for the first ca. 1 to 1.5G it performs as expected. Thereafter, however, write speed drops rapidly to ca. 25MB/s and does not recover. It is almost as if Windows translates large sequential writes to small random writes. If anyone has seen and solved this before, please let us know. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

5 9

Ceph df Vs Dashboard pool usage mismatch
by Richard Kearsley 09 Jul '20

09 Jul '20

Hi My ceph dashboard reports 64% usage for rgw.buckets.data: [image: cephdashboard.png] But "ceph df" command shows 56.81% used: RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 611 TiB 282 TiB 328 TiB 329 TiB 53.81 TOTAL 611 TiB 282 TiB 328 TiB 329 TiB 53.81 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 12 KiB 25 7.5 MiB 0 50 TiB rgw.control 5 0 B 15 0 B 0 50 TiB rgw.meta 6 13 KiB 51 14 MiB 0 50 TiB rgw.log 7 7.5 MiB 783 69 MiB 0 50 TiB rgw.buckets.index 8 57 MiB 16 57 MiB 0 50 TiB buckets.non-ec 10 0 B 0 0 B 0 50 TiB rgw.buckets.data 11 217 TiB 57.48M 328 TiB 56.81 187 TiB Does ceph df have the correct one? 🤔 ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) Thanks Richard

3 2

Octopus upgrade breaks Ubuntu 18.04 libvirt
by Andrei Mikhailovsky 08 Jul '20

08 Jul '20

Hello, I've upgraded ceph to Octopus (15.2.3 from repo) on one of the Ubuntu 18.04 host servers. The update caused problem with libvirtd which hangs when it tries to access the storage pools. The problem doesn't exist on Nautilus. The libvirtd process simply hangs. Nothing seem to happen. The log file for the libvirtd shows: 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. Running strace on the libvirtd process shows: root@ais-cloudhost1:/home/andrei# strace -p 12040 strace: Process 12040 attached restart_syscall(<... resuming interrupted poll ...> Nothing happens after that point. The same host server can get access to the ceph cluster and the pools by running ceph -s or rbd -p <pool> ls -l commands for example. Need some help to get the host servers working again with Octopus. Cheers

3 9

OSDs taking too much memory, for buffer_anon
by Harald Staub 07 Jul '20

07 Jul '20

As a follow-up to our recent memory problems with OSDs (with high pglog values: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LJPJZPBSQRJ… ), we also see high buffer_anon values. E.g. more than 4 GB, with "osd memory target" set to 3 GB. Is there a way to restrict it? As it is called "anon", I guess that it would first be necessary to find out what exactly is behind this? Well maybe it is just as Wido said, with lots of small objects, there will be several problems. Cheers Harry

4 8

Re: NFS Ganesha 2.7 in Xenial not available
by Victoria Martinez de la Cruz 03 Jul '20

03 Jul '20

Thanks Ramana and David. So we are using the Shaman search API to get the latest build for ceph_nautilus flavor of NFS Ganesha, and that's how we get to the mentioned build. We are doing this since it's part of our CI and it's better for automation. Should we use different repos? Thanks, V On Wed, Jun 24, 2020 at 3:33 PM Victoria Martinez de la Cruz < vkmc(a)redhat.com> wrote: > Thanks Ramana and David. > > So we are using the Shaman search API to get the latest build for > ceph_nautilus flavor of NFS Ganesha, and that's how we get to the mentioned > build. We are doing this since it's part of our CI and it's better for > automation. > > Should we use different repos? > > Thanks, > > V > > On Tue, Jun 23, 2020 at 2:42 PM David Galloway <dgallowa(a)redhat.com> > wrote: > >> >> >> On 6/23/20 1:21 PM, Ramana Venkatesh Raja wrote: >> > On Tue, Jun 23, 2020 at 6:59 PM Victoria Martinez de la Cruz >> > <victoria(a)redhat.com> wrote: >> >> >> >> Hi folks, >> >> >> >> I'm hitting issues with the nfs-ganesha-stable packages [0], the repo >> url >> >> [1] is broken. Is there a known issue for this? >> >> >> > >> > The missing packages in chacra could be due to the recent mishap in >> > the sepia long running cluster, >> > >> https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/YQMAHTB7MUHL25QP7V… >> >> Hi Victoria, >> >> Ramana is correct. Do you need 2.7.4 specifically? If not, signed >> nfs-ganesha packages can also be found here: >> http://download.ceph.com/nfs-ganesha/ >> >> > >> >> Thanks, >> >> >> >> Victoria >> >> >> >> [0] >> >> >> https://shaman.ceph.com/repos/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c… >> >> [1] >> >> >> https://chacra.ceph.com/r/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c1bac… >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> > >> >>

3 3

RES: Debian install
by Rafael Quaglio 02 Jul '20

02 Jul '20

Thanks for your reply Anastasios, I was waiting for some answer. My /etc/apt/sources.list.d/ceph.list content is: deb https://download.ceph.com/debian-nautilus/ buster main Even if I do “apt-get update”, the packages still the same. The Ceph client (CephFS mount) is working well, but I can´t deploy new osds. The error that I posted occurs when I do : “ceph-deploy osd create --data /dev/sdb node1” I appreciate any help. Rafael. De: Anastasios Dados <tdados(a)hotmail.com> Enviada em: segunda-feira, 29 de junho de 2020 20:01 Para: Rafael Quaglio <quaglio(a)bol.com.br>; ceph-users(a)ceph.io Assunto: Re: [ceph-users] Debian install Hello Rafael, Can you check the apt sources list that exist from your ceph-deploy node? Maybe there you have put luminous debian packages version? Regards, Anastasios On Mon, 2020-06-29 at 06:59 -0300, Rafael Quaglio wrote: Hi, We have already installed a new Debian (10.4) server and I need put it in a Ceph cluster. When I execute the command to install ceph on this node: ceph-deploy install --release nautilus node1 It starts to install a version 12.x in my node... (...) [serifos][DEBUG ] After this operation, 183 MB of additional disk space will be used. [serifos][DEBUG ] Selecting previously unselected package python-cephfs. (Reading database ... 30440 files and directories currently installed.) [serifos][DEBUG ] Preparing to unpack .../python-cephfs_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking python-cephfs (12.2.11+dfsg1-2.1+b1) ... [serifos][DEBUG ] Selecting previously unselected package ceph-common. [serifos][DEBUG ] Preparing to unpack .../ceph-common_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking ceph-common (12.2.11+dfsg1-2.1+b1) ... (...) How do I upgrade this packages? Even installed packages in this version, the installation completes without erros. The question is due to an error message that I'm recieving when deploy a new osd. ceph-deploy osd create --data /dev/sdb node1 At this point: [ceph_deploy.osd][INFO ] Distro info: debian 10.4 buster [ceph_deploy.osd][DEBUG ] Deploying osd to node1 [node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [node1][DEBUG ] find the location of an executable [node1][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [node1][WARNIN] --> RuntimeError: Unable to create a new OSD id [node1][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key [node1][DEBUG ] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 76da6c51-8385-4ffc-9a8e-0dfc11e31feb [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: In function 'void MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char>, entity_addr_t>&)' thread 7f2bc7fff700 time 2020-06-29 06:56:17.331350 [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: 77: FAILED assert(mon_info[p.first].public_addr == p.second) [node1][DEBUG ] stderr: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable) [node1][DEBUG ] stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xf5) [0x7f2bdaff5f75] [node1][DEBUG ] stderr: 2: (MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, entity_addr_t, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, entity_addr_t> > &)+0x568) [0x7f2bdb050038] [node1][DEBUG ] stderr: 3: (MonMap::decode(ceph::buffer::list::iterator&)+0x4da) [0x7f2bdb05500a] [node1][DEBUG ] stderr: 4: (MonClient::handle_monmap(MMonMap*)+0x216) [0x7f2bdb042a06] [node1][DEBUG ] stderr: 5: (MonClient::ms_dispatch(Message*)+0x4ab) [0x7f2bdb04729b] [node1][DEBUG ] stderr: 6: (DispatchQueue::entry()+0xeba) [0x7f2bdb06bf5a] [node1][DEBUG ] stderr: 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2bdb1576fd] [node1][DEBUG ] stderr: 8: (()+0x7fa3) [0x7f2be499dfa3] [node1][DEBUG ] stderr: 9: (clone()+0x3f) [0x7f2be45234cf] [node1][DEBUG ] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. [node1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs I think this error occurs because the wrong package that was installed. Thanks, Rafael _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 1

Lifecycle message on logs
by Marcelo Miziara 02 Jul '20

02 Jul '20

Hello...it's the first time I need to use the lifecycle, and I created a bucket and set it to expire in one day with s3cmd: s3cmd expire --expiry-days=1 s3://bucket The rgw_lifecycle_work_time is set to the default values(00:00-06:00). But I noticed in the rgw logs a lot of messages like: 2020-06-16 00:00:00.311369 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.8 2020-06-16 00:00:00.311623 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.311862 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.4 2020-06-16 00:00:00.319424 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.10 2020-06-16 00:00:00.319647 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.18 2020-06-16 00:00:00.320682 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.327770 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.6 2020-06-16 00:00:00.328941 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.17 2020-06-16 00:00:00.332463 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.20 2020-06-16 00:00:00.336788 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.1 2020-06-16 00:00:00.336924 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.24 2020-06-16 00:00:00.340915 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.2 The object was deleted, but these messages keep appearing. Is it safe to ignore them? For the records, i'm using redhat luminous 12.2.12 Thanks, Marcelo.

2 1

find rbd locks by client IP
by Void Star Nill 01 Jul '20

01 Jul '20

Hello, Is there a way to list all locks held by a client with the given IP address? Also, I read somewhere that removing the lock with "rbd lock rm..." automatically blacklists that client connection. Is that correct? How do I blacklist a client with the given IP address? Thanks, Shridhar

2 2

Bluestore performance tuning for hdd with nvme db+wal
by Mark Kirkwood 30 Jun '20

30 Jun '20

Hi, We have recently added a new storage node to our Luminous (12.2.13) cluster. The prev nodes are all setup as Filestore: e.g 12 osds on hdd (Seagate Constellations) with one NVMe (Intel P4600) journal. With the new guy we decided to introduce Bluestore so it is configured as: (same HW) 12 osd with data on hdd and db + wal on one NVMe. We noticed there are periodic slow requests logged, and the implicated osds are the Bluestore ones 98% of the time! This suggests that we need to tweak our Bluestore settings in some way. Investigating I'm seeing: - A great deal of rocksdb debug info in the logs - perhaps we should tone that down? (debug_rocksdb 4/5 -> 1/5) - We look to have the default cache settings (bluestore_cache_size_hdd|ssd etc), we have memory to increase these - There are some buffered io settings (bluefs_buffered_io, bluestore_default_buffered_write), set to (default) false. Are these safe (or useful) to change? - We have default rocksdb options, should some of these be changed? (bluestore_rocksdb_options, in particular max_background_compactions=2 - should we have less, or more?) Also, anything else we should be looking at? regards Mark

4 6

[RGW] Space usage vastly overestimated since Octopus upgrade
by Liam Monahan 30 Jun '20

30 Jun '20

Hi, Since upgrading from Nautilus 14.2.9 -> Octopus 15.2.3 two weeks ago we are seeing large upticks in the reported size (both space and object count) for a number of our RGW users. It does not seem to be isolated to just one user, so I don't think it's something wrong in the users' usage patterns. Users are hitting their quotas very quickly even though they are not writing anywhere near the reported space usage. Has anyone else seen this happen to them? I'm not sure what the most useful debugging information I could send would be. For example, here is a bucket that all of a sudden reports that it has 18446744073709551615 objects! The actual count should be around 20,000. [root@objproxy01 ~]# radosgw-admin bucket stats --bucket=droot-2020 { "bucket": "droot-2020", "num_shards": 32, "tenant": "", "zonegroup": "29946069-33ce-49b7-b93d-de8c95a0c344", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "8b980d5b-23de-41f9-8b14-84a5bbc3f1c9.93433056.64", "marker": "8b980d5b-23de-41f9-8b14-84a5bbc3f1c9.93433056.64", "index_type": "Normal", "owner": "-droot", "ver": "0#12052,1#15700,2#11033,3#11079,4#11521,5#13708,6#12427,7#10442,8#12769,9#11965,10#12820,11#11015,12#12073,13#11741,14#11851,15#124 97,16#10611,17#11652,18#10162,19#13699,20#9519,21#14224,22#13575,23#12635,24#9413,25#11450,26#12700,27#13122,28#10762,29#14674,30#10809,31#1223 2", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0 ,27#0,28#0,29#0,30#0,31#0", "mtime": "2020-06-29T15:14:49.363664Z", "creation_time": "2020-02-04T20:36:40.752748Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551615 }, "rgw.main": { "size": 11612169555286, "size_actual": 11612211085312, "size_utilized": 11612169555286, "size_kb": 11340009332, "size_kb_actual": 11340049888, "size_kb_utilized": 11340009332, "num_objects": 20034 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } The user who owns that bucket above is reportedly using 1.3PB of space, but the known usage is 1/10th I would guess of that until we did the upgrade. [root@objproxy01 ~]# radosgw-admin user stats --uid=-droot { "stats": { "size": 1428764900976977, "size_actual": 1428770491326464, "size_utilized": 0, "size_kb": 1395278223611, "size_kb_actual": 1395283682936, "size_kb_utilized": 0, "num_objects": 2604800 }, "last_stats_sync": "2020-06-29T13:42:26.474035Z", "last_stats_update": "2020-06-29T13:42:26.471413Z" } This seems to be happening with may users who actively write data in our Object Store. Any help appreciated! Thanks, Liam University of Maryland Institute for Advanced Computer Studies

3 4

removing the private cluster network
by Magnus HAGDORN 30 Jun '20

30 Jun '20

Hi there, we currently have a ceph cluster with 6 nodes and a public and cluster network. Each node has two bonded 2x1GE network interfaces, one for the public and one for the cluster network. We are planning to upgrade the networking to 10GE. Given the modest size of our cluster we would like to shut down the cluster network. The new 10GE switches will be on the public netowkr. What's the best way achieving this while the cluster is running. Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 1

Balancing request between rados gateway nodes
by Nghia Viet Tran 30 Jun '20

30 Jun '20

Hi everyone, We’re working on the Multi-site setup using Rados gateway with the Active-standby mode. The master zone processes all requests from the web application through the load balancer (3 Rados Gateway nodes behind). From our testing, we figured out that three Gateway nodes in backup zone makes a lot of requests into the load balancer after the period of time to check status of new objects and pulling them to the backup zone OSDs if needed. Our problem is that the workload on each Gateway nodes is not balance. Normally, one of the Gateway node is overloaded while others quite free (no incoming data, some of them even didn’t make any requests to load balancer for data checking). The load balancer from the master zone also under the high load since it’s receving data from web application and also sending data into backup zone….[cid:image001.png@01D64F0C.1252B3A0] The workload on 3 Gate ways [cid:image002.png@01D64F0C.1252B3A0] How to reduce the workload on master load balancer and balancing requests into gateway nodes in Backup zone? Many thanks! -- Nghia Viet Tran (Mr) mgm technology partners Vietnam Co. Ltd 7 Phan Châu Trinh Đà Nẵng, Vietnam +84 935905659 nghia.viet.tran(a)mgm-tp.com<mailto:nghia.viet.tran@mgm-tp.com> www.mgm-tp.com<https://www.mgm-tp.com/en/> Visit us on LinkedIn<https://www.linkedin.com/company/mgm-technology-partners-vietnam-co-ltd> and Facebook<https://www.facebook.com/mgmTechnologyPartnersVietnam>! Innovation Implemented. General Director: Frank Müller Registered office: 7 Pasteur, Hải Châu 1, Hải Châu, Đà Nẵng MST/Tax 0401703955

2 1

Re: [RGW] Space usage vastly overestimated since Octopus upgrade
by Liam Monahan 30 Jun '20

30 Jun '20

Thanks, both. That’s a useful observation. I wonder what I can try to get accurate user stats. All of our users are quota-ed, so wrong users stats actually stop them from writing data. Since stats are only updated on write: I have some users who are inactive and their stats are correct. I have other users who have been actively writing. I see users who have up to 55 times the expected vs. actual size. I looped over buckets manually via the Admin Ops API and pulled the stats for all of the user’s buckets and summed these and compared that to the output from “radosgw-admin user stats" I would guess that underflowing counters could be one explanation, but there may be other things going wrong in the stats aggregation... Thanks, Liam > On Jun 30, 2020, at 6:36 AM, EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > You can ignore rgw.none details, it dont make sense today from our experience > > Still dont know why dev dont cleanup bucket with those rgw.none stats... > > Some of our buckets got it others new ones no. > > > -----Mensaje original----- > De: Janne Johansson <icepic.dz(a)gmail.com> > Enviado el: martes, 30 de junio de 2020 8:40 > Para: Liam Monahan <liam(a)umiacs.umd.edu> > CC: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] Re: [RGW] Space usage vastly overestimated since Octopus upgrade > > Den mån 29 juni 2020 kl 17:27 skrev Liam Monahan <liam(a)umiacs.umd.edu>: > >> >> For example, here is a bucket that all of a sudden reports that it has >> 18446744073709551615 objects! The actual count should be around 20,000. >> >> "rgw.none": { >> "size": 0, >> "size_actual": 0, >> "size_utilized": 0, >> "size_kb": 0, >> "size_kb_actual": 0, >> "size_kb_utilized": 0, >> "num_objects": 18446744073709551615 >> }, >> > > That number is a small negative 64bit signed value, printed as an unsigned > 64bit integer. > Seems like the counter underflowed. > > 2^64 = 18446744073709551616 > > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Move WAL/DB to SSD for existing OSD?
by Lindsay Mathieson 30 Jun '20

30 Jun '20

Nautilus - Bluestore OSD's created with everything on disk. Now I have some spare SSD's - can I move the location of the existing WAL and/or DB to SSD partitions without recreating the OSD? I suspect not - saw emails from 2018, in the negative :( Failing that - is it difficult to add lvmcache to a osd? -- Lindsay

3 4

Best practice for object store design
by Szabo, Istvan (Agoda) 30 Jun '20

30 Jun '20

Hi, What is the let's say best practice to place haproxy, rgw, mon services in a new cluster? We would like to have a new setup, but unsure how to create a best setup in front of the OSD nodes. Let's say we have 3 mons as ceph suggest it, where should I put haproxy and rados? Should be vm or physical? Thank you the ideas. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Multisite setup with and without replicated region
by Szabo, Istvan (Agoda) 30 Jun '20

30 Jun '20

Hi, It is possible to create a multisite cluster with multiple zones? I'd like to have zone/region which is replicated across DCs, but I want to have without replication as well. Would prefer to use earlier version of ceph, not octopus yet. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Bench on specific OSD
by Seena Fallah 30 Jun '20

30 Jun '20

Hi all. Is there anyway to completely health check one OSD host or instance? For example rados bech just on that OSD or do some checks for disk and front and back netowrk? Thanks.

3 2

Issue with ceph-ansible installation, No such file or directory
by sachin.nicky＠gmail.com 30 Jun '20

30 Jun '20

Error occurs here: - name: look up for ceph-volume rejected devices ceph_volume: cluster: "{{ cluster }}" action: "inventory" register: rejected_devices environment: CEPH_VOLUME_DEBUG: 1 CEPH_CONTAINER_IMAGE: "{{ ceph_docker_registry + '/' + ceph_docker_image + ':' + ceph_docker_image_tag if containerized_deployment else None }}" CEPH_CONTAINER_BINARY: "{{ container_binary }}" PYTHONIOENCODING: utf-8 With Error: fatal: [18.225.11.17]: FAILED! => changed=false cmd: ceph-volume inventory --format=json msg: '[Errno 2] No such file or directory' rc: 2 I think ansible is not able to found the ceph-volume command. How can i fix this?

2 1

Suspicious memory leakage
by XuYun 30 Jun '20

30 Jun '20

Hi, We’ve observed some suspicious memory leak problems of MGR since upgraded to Nautilus. Yesterday I upgrade our cluster to the latest 14.2.10 and this problem seems still reproducible. According to the monitoring chart (memory usage of the active mgr node), the memory consumption started to increase with higher velocity at about 21:40pm. Up to now the reserved memory of mgr is about 8.3G according to top. I also checked the log of mgr and found that from 21:38:40, a message "client.0 ms_handle_reset on v2:10.3.1.3:6800/6” was produced every second: 2020-06-29 21:38:24.173 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9979: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 8.1 MiB/s rd, 21 MiB/s wr, 673 op/s 2020-06-29 21:38:26.180 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9980: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 9.6 MiB/s rd, 26 MiB/s wr, 764 op/s 2020-06-29 21:38:28.183 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9981: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 9.3 MiB/s rd, 23 MiB/s wr, 667 op/s 2020-06-29 21:38:30.186 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9982: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 9.3 MiB/s rd, 22 MiB/s wr, 661 op/s 2020-06-29 21:38:32.191 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9983: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 8.1 MiB/s rd, 21 MiB/s wr, 683 op/s 2020-06-29 21:38:34.195 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9984: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 4.0 MiB/s rd, 17 MiB/s wr, 670 op/s 2020-06-29 21:38:36.200 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9985: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 2.4 MiB/s rd, 15 MiB/s wr, 755 op/s 2020-06-29 21:38:38.203 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9986: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 1.2 MiB/s rd, 12 MiB/s wr, 668 op/s 2020-06-29 21:38:40.207 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9987: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 1.1 MiB/s rd, 12 MiB/s wr, 681 op/s 2020-06-29 21:38:40.887 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:41.887 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:42.213 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9988: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 1.2 MiB/s rd, 13 MiB/s wr, 735 op/s 2020-06-29 21:38:42.887 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:43.887 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:44.216 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9989: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 982 KiB/s rd, 12 MiB/s wr, 687 op/s 2020-06-29 21:38:44.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:45.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:46.222 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9990: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 1.2 MiB/s rd, 17 MiB/s wr, 789 op/s 2020-06-29 21:38:46.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:47.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:48.225 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9991: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 1.2 MiB/s rd, 15 MiB/s wr, 684 op/s 2020-06-29 21:38:48.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:49.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:50.228 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9992: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 940 KiB/s rd, 16 MiB/s wr, 674 op/s 2020-06-29 21:38:50.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:51.888 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:52.235 7f2ecab7a700 0 log_channel(cluster) log [DBG] : pgmap v9993: 1824 pgs: 3 active+clean+scrubbing+deep, 1821 active+clean; 54 TiB data, 169 TiB used, 153 TiB / 322 TiB avail; 1.3 MiB/s rd, 19 MiB/s wr, 752 op/s 2020-06-29 21:38:52.889 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 2020-06-29 21:38:53.889 7f2edbec3700 0 client.0 ms_handle_reset on v2:10.3.1.3:6800/6 I enable the debug log of another cluster which has the same problem, the log is attached Any thoughts about this issue? BR, Xu Yun

1 1

Re: Re layout help: need chassis local io to minimize net links
by Anthony D'Atri 30 Jun '20

30 Jun '20

What does “traffic” mean? Reads? Writes will have to hit the net regardless of any machinations. > On Jun 29, 2020, at 7:31 PM, Harry G. Coin <hgcoin(a)gmail.com> wrote: > > I need exactly what ceph is for a whole lot of work, that work just > doesn't represent a large fraction of the total local traffic. Ceph is > the right choice. Plainly ceph has tremendous support for replication > within a chassis, among chassis and among racks. I just need > intra-chassis traffic to not hit the net much. Seems not such an > unreasonable thing given the intra-chassis crush rules and all. After > all.. ceph's name wasn't chosen for where it can't go.... > >>>> On 6/29/20 1:57 PM, Marc Roos wrote: >> I wonder if you should not have chosen a different product? Ceph is >> meant to distribute data across nodes, racks, data centers etc. For a >> nail use a hammer, for a screw use a screw driver. >> -----Original Message----- >> To: ceph-users(a)ceph.io >> Subject: *****SPAM***** [ceph-users] layout help: need chassis local io >> to minimize net links >> Hi >> I have a few servers each with 6 or more disks, with a storage workload >> that's around 80% done entirely within each server. From a >> work-to-be-done perspective there's no need for 80% of the load to >> traverse network interfaces, the rest needs what ceph is all about. So >> I cooked up a set of crush maps and pools, one map/pool for each server >> and one map/pool for the whole. Skipping the long story, the >> performance remains network link speed bound and has got to change. >> "Chassis local" io is too slow. I even tried putting a mon within each >> server. I'd like to avoid having to revert to some other HA >> filesystem per server with ceph at the chassis layer if I can help >> it. >> Any notions that would allow 'chassis local' rbd traffic to avoid or >> mostly avoid leaving the box? >> Thanks! >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: Re layout help: need chassis local io to minimize net links
by Jeff Welling 30 Jun '20

30 Jun '20

I've not been replying to the list, apologies. > just the write metadata to the mon, with the actual write data content not having to cross a physical ethernet cable but directly to the chassis-local osds via the 'virtual' internal switch? This is my understanding as well, yes. I've not explored the ceph source yet though. On 2020-06-29 8:37 p.m., Harry G. Coin wrote: > > Jeff, thanks for the lead. When a user space rbd write has as a > destination three replica osds in the same chassis, does the whole > write get shipped out to the mon and then back, or just the write > metadata to the mon, with the actual write data content not having to > cross a physical ethernet cable but directly to the chassis-local osds > via the 'virtual' internal switch? I thought when I read the layout > of how ceph works only the control traffic goes to the mons, the data > directly from the generator to the osds. Did I get that wrong? > > > On 6/29/20 10:32 PM, Jeff W wrote: >> You mentioned setting up pools per host but still hitting network >> limits, did you try tcpdumping the NIC to see who's talking to who? >> Perhaps something isn't configured the way you expect? That may help >> you narrow down what is using the NIC as well, Mon or osd or what >> not. If it's local, I would think that the NIC wouldn't be a >> bottleneck and if it is a bottleneck I would suspect my own configs, >> but that's just my 2c. >> >> Off the top of my head im thinking it's the Mon, because even if you >> setup multiple pools I can't think of a way to have multiple groups >> of mons maintaining their own shards of consensus. Unless your >> workload is largely read only, then .. I'm not sure what the >> bottleneck would be. >> >> >> On Mon., Jun. 29, 2020, 7:32 p.m. Harry G. Coin, <hgcoin(a)gmail.com >> <mailto:hgcoin@gmail.com>> wrote: >> >> I need exactly what ceph is for a whole lot of work, that work just >> doesn't represent a large fraction of the total local traffic. >> Ceph is >> the right choice. Plainly ceph has tremendous support for >> replication >> within a chassis, among chassis and among racks. I just need >> intra-chassis traffic to not hit the net much. Seems not such an >> unreasonable thing given the intra-chassis crush rules and all. >> After >> all.. ceph's name wasn't chosen for where it can't go.... >> >> On 6/29/20 1:57 PM, Marc Roos wrote: >> > I wonder if you should not have chosen a different product? >> Ceph is >> > meant to distribute data across nodes, racks, data centers etc. >> For a >> > nail use a hammer, for a screw use a screw driver. >> > >> > >> > -----Original Message----- >> > To: ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> > Subject: *****SPAM***** [ceph-users] layout help: need chassis >> local io >> > to minimize net links >> > >> > Hi >> > >> > I have a few servers each with 6 or more disks, with a storage >> workload >> > that's around 80% done entirely within each server. From a >> > work-to-be-done perspective there's no need for 80% of the load to >> > traverse network interfaces, the rest needs what ceph is all >> about. So >> > I cooked up a set of crush maps and pools, one map/pool for >> each server >> > and one map/pool for the whole. Skipping the long story, the >> > performance remains network link speed bound and has got to >> change. >> > "Chassis local" io is too slow. I even tried putting a mon >> within each >> > server. I'd like to avoid having to revert to some other HA >> > filesystem per server with ceph at the chassis layer if I can help >> > it. >> > >> > Any notions that would allow 'chassis local' rbd traffic to >> avoid or >> > mostly avoid leaving the box? >> > >> > Thanks! >> > >> > >> > >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> <mailto:ceph-users@ceph.io> To unsubscribe send an >> > email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> >> > >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io> >>

1 0

Debian install
by Rafael Quaglio 30 Jun '20

30 Jun '20

Hi, We have already installed a new Debian (10.4) server and I need put it in a Ceph cluster. When I execute the command to install ceph on this node: ceph-deploy install --release nautilus node1 It starts to install a version 12.x in my node... (...) [serifos][DEBUG ] After this operation, 183 MB of additional disk space will be used. [serifos][DEBUG ] Selecting previously unselected package python-cephfs. (Reading database ... 30440 files and directories currently installed.) [serifos][DEBUG ] Preparing to unpack .../python-cephfs_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking python-cephfs (12.2.11+dfsg1-2.1+b1) ... [serifos][DEBUG ] Selecting previously unselected package ceph-common. [serifos][DEBUG ] Preparing to unpack .../ceph-common_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking ceph-common (12.2.11+dfsg1-2.1+b1) ... (...) How do I upgrade this packages? Even installed packages in this version, the installation completes without erros. The question is due to an error message that I'm recieving when deploy a new osd. ceph-deploy osd create --data /dev/sdb node1 At this point: [ceph_deploy.osd][INFO ] Distro info: debian 10.4 buster [ceph_deploy.osd][DEBUG ] Deploying osd to node1 [node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [node1][DEBUG ] find the location of an executable [node1][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [node1][WARNIN] --> RuntimeError: Unable to create a new OSD id [node1][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key [node1][DEBUG ] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 76da6c51-8385-4ffc-9a8e-0dfc11e31feb [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: In function 'void MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char>, entity_addr_t>&)' thread 7f2bc7fff700 time 2020-06-29 06:56:17.331350 [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: 77: FAILED assert(mon_info[p.first].public_addr == p.second) [node1][DEBUG ] stderr: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable) [node1][DEBUG ] stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xf5) [0x7f2bdaff5f75] [node1][DEBUG ] stderr: 2: (MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, entity_addr_t, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, entity_addr_t> > >&)+0x568) [0x7f2bdb050038] [node1][DEBUG ] stderr: 3: (MonMap::decode(ceph::buffer::list::iterator&)+0x4da) [0x7f2bdb05500a] [node1][DEBUG ] stderr: 4: (MonClient::handle_monmap(MMonMap*)+0x216) [0x7f2bdb042a06] [node1][DEBUG ] stderr: 5: (MonClient::ms_dispatch(Message*)+0x4ab) [0x7f2bdb04729b] [node1][DEBUG ] stderr: 6: (DispatchQueue::entry()+0xeba) [0x7f2bdb06bf5a] [node1][DEBUG ] stderr: 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2bdb1576fd] [node1][DEBUG ] stderr: 8: (()+0x7fa3) [0x7f2be499dfa3] [node1][DEBUG ] stderr: 9: (clone()+0x3f) [0x7f2be45234cf] [node1][DEBUG ] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. [node1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs I think this error occurs because the wrong package that was installed. Thanks, Rafael

4 4

Octopus missing rgw-orphan-list tool
by Andrei Mikhailovsky 29 Jun '20

29 Jun '20

Hello, I have been struggling a lot with radosgw buckets space wastage, which is currently stands at about 2/3 of utilised space is wasted and unaccounted for. I've tried to use the tools to find the orphan objects, but these were running in loop for weeks on without producing any results. Wido and a few others pointed out that this function is broke in was deprecated and that instead the rgw-orphan-list should be used instead. I have upgraded to Octopus and I have been following the documentation [ https://docs.ceph.com/docs/master/radosgw/orphans/ | https://docs.ceph.com/docs/master/radosgw/orphans/ ] . However, the ceph and radon packages for Ubuntu 18.04 do not seem to have this tool. The same applies to the bucket radoslist option to the radosgw-admin command. root@arh-ibstorage1-ib:~# radosgw-admin bucket radoslist ERROR: Unrecognized argument: 'radoslist' Expected one of the following: check chown limit link list reshard rewrite rm stats sync unlink root@arh-ibstorage1-ib:~# dpkg -l *rados\* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=======================-================-================-==================================================== un librados <none> <none> (no description available) ii librados2 15.2.3-1bionic amd64 RADOS distributed object store client library ii libradosstriper1 15.2.3-1bionic amd64 RADOS striping interface ii python3-rados 15.2.3-1bionic amd64 Python 3 libraries for the Ceph librados library ii radosgw 15.2.3-1bionic amd64 REST gateway for RADOS distributed object store I am running Ubuntu 18.04 with version 15.2.3 of ceph and radosgw. Please suggest what should I do to remove the wasted space that radosgw is creating? I've calculated the wasted space by adding up the reported usage of all the buckets and checking it agains the output of the rados df command. The buckets are using around 11TB. The rados df reports 68TB of usage with replica of 2. Rather alarming! Thanks for you help

1 0

Nautilus latest builds for CentOS 8
by Victoria Martinez de la Cruz 29 Jun '20

29 Jun '20

Hi all, I'm wondering if there is a plan to add Nautilus builds for CentOS 8 [0]. Right now I see there are builds for CentOS 7, but for CentOS 8 there are only builds for Octopus and master. Thanks, V [0] https://shaman.ceph.com/api/search/?project=ceph&distros=centos/8&ref=nauti…

6 14

Octopus Grafana inside the dashboard
by Simon Sutter 29 Jun '20

29 Jun '20

Hello I'm trying to get Grafana working inside the Dashboard. If I press on "Overall Performance" tab, I get an error, because the iframe tries to connect to the internal hostname, which cannot be resolved from my machine. If I directly open grafana, everything works. How can I tell the dashboard, to use the full domain name? I have tried to set "ceph dashboard set-grafana-api-url https://node01.mycorp.local:3000" but that does not work and always sets itself back to "https://node01:3000". Thanks in advance, Simon

2 1

Nautilus 14.2.10 mon_warn_on_pool_no_redundancy
by Wout van Heeswijk 29 Jun '20

29 Jun '20

Hi All, I really like the idea of warning users against using unsafe practices. Wouldn't it make sense to warn against using min_size=1 instead of size=1. I've seen data loss happen with size=2 min_size=1 when multiple failures occur and write have been done between the failures. Effectively the new warning below says "It is not considered safe to run with no redundancy". Which is true, but when failure occurs or maintenance is executed, with size=2 and min_size=1, as soon as data is written, there might not be data redundancy for that newly written data. A failure of an OSD at that moment would result in data loss. Since you cannot run size=1 with min_size > 1, this use-case would also be covered. I understand this has implications for size=2 when executing maintenance, but I think most people are not aware of the risks they are taking with min_size=1. Those that are aware can suppress the warning. * Ceph will issue a health warning if a RADOS pool's `size` is set to 1 or in other words the pool is configured with no redundancy. This can be fixed by setting the pool size to the minimum recommended value with:: ceph osd pool set <pool-name> size <num-replicas> The warning can be silenced with:: ceph config set global mon_warn_on_pool_no_redundancy false -- kind regards, Wout 42on

2 1

Re: rgw : unable to find part(s) of aborted multipart upload of [object].meta
by EDH - Manuel Rios 29 Jun '20

29 Jun '20

Hi Amit, Yes in the non-ec pool there’re like 600 files .meta but i dont know if its safe move to data pool. Does anyone know if there is a way to generate a synthetic .meta only so that the delete command is capable of deleting the file? Regards Manuel De: Amit Ghadge <amitg.b14(a)gmail.com> Enviado el: lunes, 29 de junio de 2020 6:14 Para: EDH - Manuel Rios <mriosfer(a)easydatahost.com> Asunto: Re: [ceph-users] rgw : unable to find part(s) of aborted multipart upload of [object].meta You can also check in default.rgw.buckets.non-ec pool for unmerged multipart or bucket indexes that need to be fix. On Mon, Jun 29, 2020 at 5:56 AM EDH - Manuel Rios <mriosfer(a)easydatahost.com<mailto:mriosfer@easydatahost.com>> wrote: Hi Dev's With the failures of the previous versions in the buckets due to the shardings. We have started a copy of the buckets to new buckets to clean our ceph cluster. After synchronizing the bucket with the AWS cli, we are in the phase of deleting the old buckets. We have tried unsuccessfully: radosgw-admin bucket rm --bucket = XXXX --purge-objects Result is a loop that shows with the debug: "NOTE: unable to find part (s) of aborted multipart upload of [object] .meta" After seeing this failure we have tried to clean up using "rados". For this we have listed all the objects belonging to the pool bucket using its marker_id. Once done, we send them a script to delete massively, with "rados -p [rgw-pool.data] rm [object]" The result of all is similar to the following: rados -p default.rgw.buckets.data rm 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc error removing default.rgw.buckets.data>48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc: (2) No such file or directory Any idea or better way to clean the cluster of these objects? Our estimates are 100TB of wrong objects. Regards Manuel _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

mgr log shows a lot of ms_handle_reset messages
by XuYun 29 Jun '20

29 Jun '20

Hi, We are running Ceph nautilus (14.2.10 now), every mgr reports a ‘ms_handle_reset’ message every second. Would it be a normal behavior? 2020-06-28 16:35:43.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:44.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:45.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:46.581 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 2020-06-28 16:35:47.580 7fcdc96e6700 0 client.0 ms_handle_reset on v2:111.111.121.3:6807/7 Br, Xu Yun

2 4

rgw : unable to find part(s) of aborted multipart upload of [object].meta
by EDH - Manuel Rios 29 Jun '20

29 Jun '20

Hi Dev's With the failures of the previous versions in the buckets due to the shardings. We have started a copy of the buckets to new buckets to clean our ceph cluster. After synchronizing the bucket with the AWS cli, we are in the phase of deleting the old buckets. We have tried unsuccessfully: radosgw-admin bucket rm --bucket = XXXX --purge-objects Result is a loop that shows with the debug: "NOTE: unable to find part (s) of aborted multipart upload of [object] .meta" After seeing this failure we have tried to clean up using "rados". For this we have listed all the objects belonging to the pool bucket using its marker_id. Once done, we send them a script to delete massively, with "rados -p [rgw-pool.data] rm [object]" The result of all is similar to the following: rados -p default.rgw.buckets.data rm 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc error removing default.rgw.buckets.data>48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.18__multipart_MBS-3369403d-e0bf-45e3-89ba-614b6d390dc5/CBB_BIM-EURODG/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20200104230152/131.cbrevision.5K5_leiUZoHQsjBvUxw2QbM1WQPQLlc: (2) No such file or directory Any idea or better way to clean the cluster of these objects? Our estimates are 100TB of wrong objects. Regards Manuel

1 0

fault tolerant about erasure code pool
by Zhenshi Zhou 28 Jun '20

28 Jun '20

Hi all, I'm going to deploy a cluster with erasure code pool for cold storage. There are 3 servers for me to set up the cluster, 12 OSDs on each server. Does that mean the data is secure while 1/3 OSDs of the cluster is down, or only 2 of the OSDs is down , if I set the ec profile with k=4 and m=2. Thanks

7 12

High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs
by Benoît Knecht 27 Jun '20

27 Jun '20

Hi, We have a Nautilus (14.2.9) Ceph cluster with two types of HDDs: - TOSHIBA MG07ACA14TE [1] - HGST HUH721212ALE604 [2] They're all bluestore OSDs with no separate DB+WAL and part of the same pool. We noticed that while the HGST OSDs have a commit latency of about 15ms, the Toshiba OSDs hover around 150ms (these values come from the `ceph_osd_commit_latency_ms` metric in Prometheus). On paper, it seems like those drives have very similar specs, so it's not clear to me why we're seeing such a large difference when it comes to commit latency. Has anyone had any experience with those Toshiba drives? Or looking at the specs, do you spot anything suspicious? And if you're running a Ceph cluster with various disk brands/models, have you ever noticed some of them standing out when looking at `ceph_osd_commit_latency_ms`? Thanks in advance for your feedback. Cheers, -- Ben [1]: https://toshiba.semicon-storage.com/content/dam/toshiba-ss/asia-pacific/doc… [2]: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/p…

15 27

bluestore_throttle_bytes
by Seena Fallah 27 Jun '20

27 Jun '20

Hi all. Can someone explain or reference a doc about the bluestore_throttle_bytes option? I don't find any docs for this config. Thanks.

1 0

CephFS: What is the maximum number of files per directory
by Martin Palma 27 Jun '20

27 Jun '20

Hi, What is the maximum number of files per directory? I could find the answer in the docs. Best, Martin

4 3

re Centos8 / octopus installation question
by Mazzystr 26 Jun '20

26 Jun '20

Can anyone explain why ktdryer's dev repo is still landing on production installs? [root@centos8 ~]# ./cephadm add-repo --release octopus INFO:root:Writing repo to /etc/yum.repos.d/ceph.repo... INFO:cephadm:Enabling EPEL... INFO:cephadm:Enabling supplementary copr repo ktdreyer/ceph-el8... [root@centos8 ~]# ls /etc/yum.repos.d/ CentOS-AppStream.repo CentOS-Debuginfo.repo CentOS-Media.repo ceph.repo epel.repo CentOS-Base.repo CentOS-Extras.repo CentOS-PowerTools.repo _copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8.repo epel-testing-modular.repo CentOS-centosplus.repo CentOS-fasttrack.repo CentOS-Sources.repo epel-modular.repo epel-testing.repo CentOS-CR.repo CentOS-HA.repo CentOS-Vault.repo epel-playground.repo [root@centos8 ~]# cat /etc/yum.repos.d/_copr\:copr.fedorainfracloud.org \:ktdreyer\:ceph-el8.repo [copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8] name=Copr repo for ceph-el8 owned by ktdreyer baseurl= https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/epel-8… type=rpm-md skip_if_unavailable=True gpgcheck=1 gpgkey= https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey… repo_gpgcheck=0 enabled=1 enabled_metadata=1 Thanks, /Chris C

1 0

Pointers in __crush_do_rule__ function of CRUSH mapper file
by Bobby 26 Jun '20

26 Jun '20

Hi all, I have a question regarding pointer variables used in the __crush_do_rule__ function of CRUSH __mapper.c__. Can someone please help me understand the purpose of following four pointer variables inside __crush_do_rule__: int *b = a + result_max; int *c = b + result_max; int *w = a; int *o = b; The function __crush_do_rule__ is below: /** * crush_do_rule - calculate a mapping with the given input and rule * @map: the crush_map * @ruleno: the rule id * @x: hash input * @result: pointer to result vector * @result_max: maximum result size * @weight: weight vector (for map leaves) * @weight_max: size of weight vector * @cwin: Pointer to at least map->working_size bytes of memory or NULL. */ int crush_do_rule(const struct crush_map *map, int ruleno, int x, int *result, int result_max, const __u32 *weight, int weight_max, void *cwin, const struct crush_choose_arg *choose_args) { int result_len; struct crush_work *cw = cwin; int *a = (int *)((char *)cw + map->working_size); int *b = a + result_max; int *c = b + result_max; int *w = a; int *o = b; int recurse_to_leaf; int wsize = 0; int osize; int *tmp; const struct crush_rule *rule; __u32 step; int i, j; int numrep; int out_size; Thanks Bobby !

1 0

v14.2.10 Nautilus released
by Abhishek Lekshmanan 26 Jun '20

26 Jun '20

We're happy to announce the tenth release in the Nautilus series. In addition to fixing a security-related bug in RGW, this release brings a number of bugfixes across all major components of Ceph. We recommend that all Nautilus users upgrade to this release. For a detailed changelog please refer to the ceph release blog at: https://ceph.io/releases/v14-2-10-nautilus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * RGW: Bucket notifications now support Kafka endpoints. This requires librdkafka of version 0.9.2 and up. Note that Ubuntu 16.04.6 LTS (Xenial Xerus) has an older version of librdkafka, and would require an update to the library. * The pool parameter `target_size_ratio`, used by the pg autoscaler, has changed meaning. It is now normalized across pools, rather than specifying an absolute ratio. For details, see :ref:`pg-autoscaler`. If you have set target size ratios on any pools, you may want to set these pools to autoscale `warn` mode to avoid data movement during the upgrade:: ceph osd pool set <pool-name> pg_autoscale_mode warn * The behaviour of the `-o` argument to the rados tool has been reverted to its orignal behaviour of indicating an output file. This reverts it to a more consistent behaviour when compared to other tools. Specifying object size is now accomplished by using an upper case O `-O`. * The format of MDSs in `ceph fs dump` has changed. * Ceph will issue a health warning if a RADOS pool's `size` is set to 1 or in other words the pool is configured with no redundancy. This can be fixed by setting the pool size to the minimum recommended value with:: ceph osd pool set <pool-name> size <num-replicas> The warning can be silenced with:: ceph config set global mon_warn_on_pool_no_redundancy false * RGW: bucket listing performance on sharded bucket indexes has been notably improved by heuristically -- and significantly, in many cases -- reducing the number of entries requested from each bucket index shard. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: b340acf629a010a74d90da5782a2c5fe0b54ac20 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

osd init authentication failed: (1) Operation not permitted
by Naumann, Thomas 26 Jun '20

26 Jun '20

Hi, in our production cluster (proxmox 5.4, ceph 12.2) there is an issue since yesterday. after an increase of a pool 5 OSDs do not start, status is "down/in", ceph health: HEALTH_WARN nodown,noout flag(s) set, 5 osds down, 128 osds: 123 up, 128 in. last lines of OSD-logfile: 2020-06-26 08:40:26.240005 7f6d245fff80 1 freelist init 2020-06-26 08:40:26.243779 7f6d245fff80 1 bluestore(/var/lib/ceph/osd/ceph-45) _open_alloc opening allocation metadata 2020-06-26 08:40:26.251501 7f6d245fff80 1 bluestore(/var/lib/ceph/osd/ceph-45) _open_alloc loaded 3.47TiB in 1 extents 2020-06-26 08:40:26.253058 7f6d245fff80 0 <cls> /mnt/big/pve/ceph/ceph-12.2.13/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2020-06-26 08:40:26.253309 7f6d245fff80 0 _get_class not permitted to load sdk 2020-06-26 08:40:26.256486 7f6d245fff80 0 _get_class not permitted to load kvs 2020-06-26 08:40:26.256611 7f6d245fff80 0 <cls> /mnt/big/pve/ceph/ceph-12.2.13/src/cls/hello/cls_hello.cc:296: loading cls_hello 2020-06-26 08:40:26.258362 7f6d245fff80 0 _get_class not permitted to load lua 2020-06-26 08:40:26.259850 7f6d245fff80 0 osd.45 46770 crush map has features 288514051259236352, adjusting msgr requires for clients 2020-06-26 08:40:26.259859 7f6d245fff80 0 osd.45 46770 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons 2020-06-26 08:40:26.259863 7f6d245fff80 0 osd.45 46770 crush map has features 1009089991638532096, adjusting msgr requires for osds 2020-06-26 08:40:26.305880 7f6d245fff80 0 osd.45 46770 load_pgs 2020-06-26 08:40:28.024638 7f6d245fff80 0 osd.45 46770 load_pgs opened 129 pgs 2020-06-26 08:40:28.024803 7f6d245fff80 0 osd.45 46770 using weightedpriority op queue with priority op cut off at 64. 2020-06-26 08:40:28.025741 7f6d245fff80 -1 osd.45 46770 log_to_monitors {default=true} 2020-06-26 08:40:28.028397 7f6d245fff80 -1 osd.45 46770 init authentication failed: (1) Operation not permitted Does anyone know how to fix this? -- Thomas Naumann Abteilung Netze und Kommunikation Otto-von-Guericke Universität Magdeburg Universitätsrechenzentrum Universitätsplatz 2 39106 Magdeburg fon: +49 391 67-58563 email: thomas.naumann(a)ovgu.de

2 1

Removing pool in nautilus is incredibly slow
by Francois Legrand 26 Jun '20

26 Jun '20

Hello, I am running ceph nautilus 14.2.8 I had to remove 2 pools (old cephfs data and metadata pool with 1024 pgs). The removal of the pools seems to take a incredible time to free the space (the data pool I deleted was more than 100 TB and in 36h I got back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~1h30 mn for a 32 GB image and writing 10MB in cephfs takes half a minute !!) which makes the cluster almost unusable. It seems that the removal of deleted pg is done by deep-scrubs according tohttps://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301 Also it has been reported that this could be a regression in nautilushttps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXFGJGDYZALG6TQ4QBVGGAJ/#W4M5XQRDBLXFGJGDYZALG6TQ4QBVGGAJ But I couldn't find a fix or a way to speedup (or slow down) the process and get back the cluster to a decent reactivity. Is there a way ? Thanks F.

7 21

ceph qos
by 展荣臻（信泰） 26 Jun '20

26 Jun '20

Hi everyone: There are two types of qos in ceph(one based on tokenbucket algorithm,another based on mclock ). Which one I can use in nautilus production environment ？Thank you

2 1

Re: Ceph Tech Talk: Solving the Bug of the Year
by Dan van der Ster 26 Jun '20

26 Jun '20

Hi Marc, None of the CephFS issues are show-stoppers but we're anyway waiting for them to land in nautilus: * https://tracker.ceph.com/issues/45090 * https://tracker.ceph.com/issues/45261 * https://tracker.ceph.com/issues/45835 * https://tracker.ceph.com/issues/45875 Cheers, Dan On Thu, Jun 25, 2020 at 11:38 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Top! Good to see such pro's on the team. > > What bugs is Dan waiting for to be fixed in cephfs before he upgrades > from luminous to nautilus? > > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] Ceph Tech Talk: Solving the Bug of the Year > > Hi everyone, > > Thanks again to everyone who was able to join us for discussion, and to > Dan for providing some great content. You can find the full recording > for the latest Ceph Tech Talk here: > > https://www.youtube.com/watch?v=_4HUR00oCGo > > We're looking for a talk for August 27th. If you're available and have > content to share, let me know! > > -- > > Mike Perez > > He/Him > > Ceph Community Manager > > Red Hat Los Angeles <https://www.redhat.com> > > thingee(a)redhat.com <mailto:thingee@redhat.com> > M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee > > 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA > > @Thingee <https://twitter.com/thingee> > <https://www.redhat.com> > <https://redhat.com/summit> > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

RGW listing slower on nominally faster setup
by Stefan Wild 26 Jun '20

26 Jun '20

Hi everyone, We are currently transitioning from a temporary machine to our production hardware. Since we're starting with under 200 TB raw storage, we are currently on only 1–2 physical machines per cluster, eventually in 3 zones. The temporary machine is undersized for even that with an older single 6-core CPU and spinning disks only. As of now that "cluster-of-one" is running on Nautilus and has 3 buckets with 98K, 1.1M and 1.4M objects, respectively for a total of 9.1 TB. As we're expecting these to grow to around 5M objects each and will be in a multisite configuration, I went with 50 shards per bucket. Listing "directories" via S3 is somewhat slow (sometimes to the point of read timeouts) but mostly bearable. After the new production setup (dual 8-core/16-thread Xeon Silvers, 2 x SATA SSDs for RGW index pool, on Octopus, with enough free memory to easily fit all bucket indexes multiple times) synced successfully, listings via S3 always time out on the RGW on that machine/zone. As soon as I trigger a single listing via S3 (even on the 98K object bucket), reads go up to a sustained 300–500MB/s and 20–50K IOPS on the bucket index pool for several hours. The RGW debug log is flooded with lines like this: {"log":"debug 2020-06-08T19:31:08.315+0000 7f83d704c700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1\n","stream":"stdout","time":"2020-06-08T19:31:08.317198682Z"} I get that sharded RGW indexes (and listing objects in S3 buckets in general) are not very efficient, but after getting somewhat decent results on slower hardware and an older Ceph version, I wasn't expecting the nominally much better setup to be orders of magnitude slower. Any help or pointers would be greatly appreciated. Thank you, Stefan

7 8

Re: Removing pool in nautilus is incredibly slow
by Chad William Seys 25 Jun '20

25 Jun '20

Do you mean unfound instead of undersized? There is an as yet unreproducible bug: https://tracker.ceph.com/issues/44286 (Please follow this bug if it affects you! I've experienced it and am leery of doing any drive swaps or upgrades until it is fixed.) Chad.

1 0

Ceph Tech Talk: Solving the Bug of the Year
by Mike Perez 25 Jun '20

25 Jun '20

Hi everyone, Thanks again to everyone who was able to join us for discussion, and to Dan for providing some great content. You can find the full recording for the latest Ceph Tech Talk here: https://www.youtube.com/watch?v=_4HUR00oCGo We're looking for a talk for August 27th. If you're available and have content to share, let me know! -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

2 1

node-exporter error problem
by Cem Zafer 25 Jun '20

25 Jun '20

Hi, Our ceph cluster system health is fine but when I looked at the "ceph orch ps" one of the image has error state as stated below. node-exporter.ceph102 ceph102 error 7m ago 13m <unknown> prom/node-exporter <unknown> <unknown> How can we debug and locate the problem with ceph command? Another thing is where can I find that error log? Inside the docker or host? Regards.

1 0

Unsubscribe this mail list
by adan 25 Jun '20

25 Jun '20

hello i want to unsubscribe this mail list . help me please. 在 2020/6/25 22:42, ceph-users-request(a)ceph.io 写道: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: Removing pool in nautilus is incredibly slow > (Francois Legrand) > 2. Re: Bench on specific OSD (Marc Roos) > 3. Re: Removing pool in nautilus is incredibly slow > (Wout van Heeswijk) > 4. Lifecycle message on logs (Marcelo Miziara) > 5. Re: Feedback of the used configuration (Simon Sutter) > 6. Re: Removing pool in nautilus is incredibly slow > (Francois Legrand) > 7. Re: Removing pool in nautilus is incredibly slow (Eugen Block) > > > ---------------------------------------------------------------------- > > Date: Thu, 25 Jun 2020 07:57:48 -0000 > From: "Francois Legrand" <fleg(a)lpnhe.in2p3.fr> > Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow > To: ceph-users(a)ceph.io > Message-ID: <159307186843.20.7354239610800797635@mailman-web> > Content-Type: text/plain; charset="utf-8" > > Does someone have an idea ? > F. > > ------------------------------ > > Date: Thu, 25 Jun 2020 11:36:24 +0200 > From: "Marc Roos" <M.Roos(a)f1-outsourcing.eu> > Subject: [ceph-users] Re: Bench on specific OSD > To: ceph-users <ceph-users(a)ceph.io>, seenafallah > <seenafallah(a)gmail.com> > Message-ID: <"H0000071001729d1.1593077784.sx.f1-outsourcing.eu*"@MHS> > Content-Type: text/plain; charset="US-ASCII" > > > > What is wrong with just doing multiple tests and group in your charts > osd's by host? > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] Bench on specific OSD > > Hi all. > > Is there anyway to completely health check one OSD host or instance? > For example rados bech just on that OSD or do some checks for disk and > front and back netowrk? > > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > > ------------------------------ > > Date: Thu, 25 Jun 2020 14:26:59 +0200 > From: Wout van Heeswijk <wout(a)42on.com> > Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow > To: fleg(a)lpnhe.in2p3.fr > Cc: ceph-users(a)ceph.io > Message-ID: <38a5d557-7797-a68a-1b67-4c8f0e1ecf4c(a)42on.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi Francois, > > Have you already looked at the option "osd_delete_sleep"? It will not > speed up the process but I will give you some control over your cluster > performance. > > Something like: > > ceph tell osd.\* injectargs '--osd_delete_sleep1' > > kind regards, > > Wout > 42on > > On 25-06-2020 09:57, Francois Legrand wrote: >> Does someone have an idea ? >> F. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Thu, 25 Jun 2020 09:53:09 -0300 > From: Marcelo Miziara <raxidex(a)gmail.com> > Subject: [ceph-users] Lifecycle message on logs > To: ceph-users(a)ceph.io > Message-ID: > <CAGxsqB296eczKux19OKObkimXg3PZ_UDEWVTTqEnrAf=Agm0+A(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hello...it's the first time I need to use the lifecycle, and I created a > bucket and set it to expire in one day with s3cmd: > s3cmd expire --expiry-days=1 s3://bucket > > The rgw_lifecycle_work_time is set to the default values(00:00-06:00). But > I noticed in the rgw logs a lot of messages like: > 2020-06-16 00:00:00.311369 7fe2cac87700 0 RGWLC::process() failed to get > obj entry lc.8 > 2020-06-16 00:00:00.311623 7fe2c8c83700 0 RGWLC::process() failed to get > obj entry lc.16 > 2020-06-16 00:00:00.311862 7fe2c6c7f700 0 RGWLC::process() failed to get > obj entry lc.4 > 2020-06-16 00:00:00.319424 7fe2cac87700 0 RGWLC::process() failed to get > obj entry lc.10 > 2020-06-16 00:00:00.319647 7fe2c8c83700 0 RGWLC::process() failed to get > obj entry lc.18 > 2020-06-16 00:00:00.320682 7fe2c6c7f700 0 RGWLC::process() failed to get > obj entry lc.16 > 2020-06-16 00:00:00.327770 7fe2cac87700 0 RGWLC::process() failed to get > obj entry lc.6 > 2020-06-16 00:00:00.328941 7fe2c8c83700 0 RGWLC::process() failed to get > obj entry lc.17 > 2020-06-16 00:00:00.332463 7fe2c6c7f700 0 RGWLC::process() failed to get > obj entry lc.20 > 2020-06-16 00:00:00.336788 7fe2cac87700 0 RGWLC::process() failed to get > obj entry lc.1 > 2020-06-16 00:00:00.336924 7fe2c8c83700 0 RGWLC::process() failed to get > obj entry lc.24 > 2020-06-16 00:00:00.340915 7fe2c6c7f700 0 RGWLC::process() failed to get > obj entry lc.2 > > The object was deleted, but these messages keep appearing. > Is it safe to ignore them? > > For the records, i'm using redhat luminous 12.2.12 > > Thanks, Marcelo. > > ------------------------------ > > Date: Thu, 25 Jun 2020 13:19:38 +0000 > From: Simon Sutter <ssutter(a)hosttech.ch> > Subject: [ceph-users] Re: Feedback of the used configuration > To: Paul Emmerich <paul.emmerich(a)croit.io> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <f6e11ee189a74c35815c495d64cbe4cb(a)hosttech.ch> > Content-Type: text/plain; charset="utf-8" > > Hello Paul, > > Thanks for the Answer. > I took a look at the subvolumes, but they are a bit odd in my opinion. > If I create one with a subvolume-group, the folder structure will look like this: > /cephfs/volumes/group-name/subvolume-name/random-uuid/ > And I have to issue two commands, first set the group and then set the volume name, but why so complicated? > > Wouldn't it be easier to just make subvolumes anywhere inside the cephfs? > I can see the intended use for groups, but if I want to publish a pool in some different directory that's not possible (except for setfattr). > Without first creating subvolume-groups, the orchestrator creates subvolumes in the /cephfs/volumes/_nogroup/subvolume-name/randmon-uuid/ folder. > > And the more important question is, why is there a new folder with a random uuid inside the subvolume? > I try to understand the points the devs had, when they developed this, but for me, this is something I have to explain to some devs in our team and at the moment I can't. > > It is indeed easier to deploy but comes with much less flexibility. > Maybe something to write in the tracker about? > > Thanks in advance, > Simon > > Von: Paul Emmerich [mailto:paul.emmerich@croit.io] > Gesendet: Mittwoch, 24. Juni 2020 17:35 > An: Simon Sutter <ssutter(a)hosttech.ch> > Cc: ceph-users(a)ceph.io > Betreff: Re: [ceph-users] Feedback of the used configuration > > Have a look at cephfs subvolumes: https://docs.ceph.com/docs/master/cephfs/fs-volumes/#fs-subvolumes > > They are internally just directories with quota/pool placement layout/namespace with some mgr magic to make it easier than doing that all by hand > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io<http://www.croit.io> > Tel: +49 89 1896585 90 > > > On Wed, Jun 24, 2020 at 4:38 PM Simon Sutter <ssutter(a)hosttech.ch<mailto:ssutter@hosttech.ch>> wrote: > Hello, > > After two months of the "ceph try and error game", I finally managed to get an Octopuss cluster up and running. > The unconventional thing about it is, it's just for hot backups, no virtual machines on there. > All the nodes are without any caching ssd's, just plain hdd's. > At the moment there are eight of them with a total of 50TB. We are planning to go up to 25 and bigger disks so we end on 300TB-400TB > > I decided to go with cephfs, because I don't have any experience in things like S3 and I need to read the same file system from more than one client. > > I made one cephfs with a replicated pool. > On there I added erasure-coded pools to save some Storage. > To add those pools, I did it with the setfattr command like this: > setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1 > > Some of our servers cannot use cephfs (old kernels, special OS's) so I have to use nfs. > This is set up with the included ganesha-nfs. > Exported is the /cephfs/nfs folder and clients can mount folders below this. > > There are two final questions: > > - Was it right to go with the way of "mounting" pools with setfattr, or should I have used multiple cephfs? > > First I was thinking about using multiple cephfs but there are warnings everywhere. The deeper I got in, the more it seems I would have been fine with multiple cephfs. > > - Is there a way I don't know, but it would be easier? > > I still don't know much about Rest, S3, RBD etc... so there may be a better way > > Other remarks are desired. > > Thanks in advance, > Simon > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > > ------------------------------ > > Date: Thu, 25 Jun 2020 16:22:12 +0200 > From: Francois Legrand <fleg(a)lpnhe.in2p3.fr> > Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow > To: Wout van Heeswijk <wout(a)42on.com> > Cc: ceph-users(a)ceph.io > Message-ID: <db34e791-1172-8b94-09e7-4ab5790d3162(a)lpnhe.in2p3.fr> > Content-Type: text/plain; charset=utf-8; format=flowed > > Thanks for the hint. > I tryed but it doesn't seems to change anything... > Moreover, as the osds seems quite loaded I had regularly some osd marked > down which triggered some new peering and thus more load !!! > I set the osd no down flag, but I still have some osd reported (wrongly) > as down (and back up in the minute) which generate peering and > remapping. I don't really understand the action of no down parameter ! > Is there a way to tell ceph not to peer immediately after an osd is > reported down (let say wait for 60s) ? > I am thinking about restarting all osd (or maybe the whole cluster) to > get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to > something higher than 15 (but I don't think it will really improve the > situation). > F. > > > Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit : >> Hi Francois, >> >> Have you already looked at the option "osd_delete_sleep"? It will not >> speed up the process but I will give you some control over your >> cluster performance. >> >> Something like: >> >> ceph tell osd.\* injectargs '--osd_delete_sleep1' >> kind regards, >> >> Wout >> 42on >> On 25-06-2020 09:57, Francois Legrand wrote: >>> Does someone have an idea ? >>> F. >>> _______________________________________________ >>> ceph-users mailing list --ceph-users(a)ceph.io >>> To unsubscribe send an email toceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Thu, 25 Jun 2020 14:42:57 +0000 > From: Eugen Block <eblock(a)nde.ag> > Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow > To: ceph-users(a)ceph.io > Message-ID: > <20200625144257.Horde.I8JPcddeor47WdOehQKQPNY(a)webmail.nde.ag> > Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes > > I'm not sure if your OSDs have their rocksDB on faster devices, if not > it sounds a lot like rocksdb fragmentation [1] leading to a very high > load on the OSDs and occasionally crashing OSDs. If you don't plan to > delete so much data at once on a regular basis you could sit this one > out, but one solution is to re-create the OSDs with rocksDB/WAL on > faster devices. > > > [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html > > > Zitat von Francois Legrand <fleg(a)lpnhe.in2p3.fr>: > >> Thanks for the hint. >> I tryed but it doesn't seems to change anything... >> Moreover, as the osds seems quite loaded I had regularly some osd >> marked down which triggered some new peering and thus more load !!! >> I set the osd no down flag, but I still have some osd reported >> (wrongly) as down (and back up in the minute) which generate peering >> and remapping. I don't really understand the action of no down >> parameter ! >> Is there a way to tell ceph not to peer immediately after an osd is >> reported down (let say wait for 60s) ? >> I am thinking about restarting all osd (or maybe the whole cluster) >> to get osd_op_queue_cut_off changed to high and >> osd_op_thread_timeout to something higher than 15 (but I don't think >> it will really improve the situation). >> F. >> >> >> Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit : >>> Hi Francois, >>> >>> Have you already looked at the option "osd_delete_sleep"? It will >>> not speed up the process but I will give you some control over your >>> cluster performance. >>> >>> Something like: >>> >>> ceph tell osd.\* injectargs '--osd_delete_sleep1' >>> kind regards, >>> >>> Wout >>> 42on >>> On 25-06-2020 09:57, Francois Legrand wrote: >>>> Does someone have an idea ? >>>> F. >>>> _______________________________________________ >>>> ceph-users mailing list --ceph-users(a)ceph.io >>>> To unsubscribe send an email toceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 89, Issue 112 > *******************************************

2 1

Re: Feedback of the used configuration
by Simon Sutter 25 Jun '20

25 Jun '20

Hello Paul, Thanks for the Answer. I took a look at the subvolumes, but they are a bit odd in my opinion. If I create one with a subvolume-group, the folder structure will look like this: /cephfs/volumes/group-name/subvolume-name/random-uuid/ And I have to issue two commands, first set the group and then set the volume name, but why so complicated? Wouldn't it be easier to just make subvolumes anywhere inside the cephfs? I can see the intended use for groups, but if I want to publish a pool in some different directory that's not possible (except for setfattr). Without first creating subvolume-groups, the orchestrator creates subvolumes in the /cephfs/volumes/_nogroup/subvolume-name/randmon-uuid/ folder. And the more important question is, why is there a new folder with a random uuid inside the subvolume? I try to understand the points the devs had, when they developed this, but for me, this is something I have to explain to some devs in our team and at the moment I can't. It is indeed easier to deploy but comes with much less flexibility. Maybe something to write in the tracker about? Thanks in advance, Simon Von: Paul Emmerich [mailto:paul.emmerich@croit.io] Gesendet: Mittwoch, 24. Juni 2020 17:35 An: Simon Sutter <ssutter(a)hosttech.ch> Cc: ceph-users(a)ceph.io Betreff: Re: [ceph-users] Feedback of the used configuration Have a look at cephfs subvolumes: https://docs.ceph.com/docs/master/cephfs/fs-volumes/#fs-subvolumes They are internally just directories with quota/pool placement layout/namespace with some mgr magic to make it easier than doing that all by hand Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90 On Wed, Jun 24, 2020 at 4:38 PM Simon Sutter <ssutter(a)hosttech.ch<mailto:ssutter@hosttech.ch>> wrote: Hello, After two months of the "ceph try and error game", I finally managed to get an Octopuss cluster up and running. The unconventional thing about it is, it's just for hot backups, no virtual machines on there. All the nodes are without any caching ssd's, just plain hdd's. At the moment there are eight of them with a total of 50TB. We are planning to go up to 25 and bigger disks so we end on 300TB-400TB I decided to go with cephfs, because I don't have any experience in things like S3 and I need to read the same file system from more than one client. I made one cephfs with a replicated pool. On there I added erasure-coded pools to save some Storage. To add those pools, I did it with the setfattr command like this: setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1 Some of our servers cannot use cephfs (old kernels, special OS's) so I have to use nfs. This is set up with the included ganesha-nfs. Exported is the /cephfs/nfs folder and clients can mount folders below this. There are two final questions: - Was it right to go with the way of "mounting" pools with setfattr, or should I have used multiple cephfs? First I was thinking about using multiple cephfs but there are warnings everywhere. The deeper I got in, the more it seems I would have been fine with multiple cephfs. - Is there a way I don't know, but it would be easier? I still don't know much about Rest, S3, RBD etc... so there may be a better way Other remarks are desired. Thanks in advance, Simon _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Feedback of the used configuration
by Simon Sutter 24 Jun '20

24 Jun '20

Hello, After two months of the "ceph try and error game", I finally managed to get an Octopuss cluster up and running. The unconventional thing about it is, it's just for hot backups, no virtual machines on there. All the nodes are without any caching ssd's, just plain hdd's. At the moment there are eight of them with a total of 50TB. We are planning to go up to 25 and bigger disks so we end on 300TB-400TB I decided to go with cephfs, because I don't have any experience in things like S3 and I need to read the same file system from more than one client. I made one cephfs with a replicated pool. On there I added erasure-coded pools to save some Storage. To add those pools, I did it with the setfattr command like this: setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1 Some of our servers cannot use cephfs (old kernels, special OS's) so I have to use nfs. This is set up with the included ganesha-nfs. Exported is the /cephfs/nfs folder and clients can mount folders below this. There are two final questions: - Was it right to go with the way of "mounting" pools with setfattr, or should I have used multiple cephfs? First I was thinking about using multiple cephfs but there are warnings everywhere. The deeper I got in, the more it seems I would have been fine with multiple cephfs. - Is there a way I don't know, but it would be easier? I still don't know much about Rest, S3, RBD etc... so there may be a better way Other remarks are desired. Thanks in advance, Simon

2 1

How to ceph-volume on remote hosts?
by steven prothero 24 Jun '20

24 Jun '20

Hello, I am new to CEPH and on a few test servers attempting to setup and learn a test ceph system. I started off the install with the "Cephadm" option and it uses podman containers. Followed steps here: https://docs.ceph.com/docs/master/cephadm/install/ I ran the bootstrap, added remote hosts, added monitors and everything is looking good. Now I would like to add OSDs... On the bootstrapped server i did a : ceph-volume lvm prepare --data /dev/sda6 and then the "activate" and "ceph orch daemon add osd (etc)" to add it and it works... But now I am ready to add OSDs on the remote nodes. I am not able to find documentation or examples on how to do : ceph-volume lvm prepare & activate steps on the remote hosts. How do we prepare & activate the remote hosts disks? Thank you very much for your input, Cheers Steve

3 2

How to remove one of two filesystems
by Francois Legrand 24 Jun '20

24 Jun '20

Hello, I have a ceph cluster (nautilus 14.2.8) with 2 filesystems and 3 mds. mds1 is managing fs1 mds2 manages fs2 mds3 is standby I want to completely remove fs1. It seems that the command to use is ceph fs rm fs1 --yes-i-really-mean-it and then delete the data and metadata pools with ceph osd pool delete but in many threads I noticed that you must shutdown the mds before running ceph fs rm. Is it still the case ? What happens in my configuration (I have 2 fs) ? If I stop mds1, the mds3 will take the management. If I stop mds3 what will mds2 do (try to manage the 2 fs or continue only with fs2) ? Thanks for your advices. F.

4 7

Nautilus: Monitors not listening on msgrv1
by Julian Fölsch 23 Jun '20

23 Jun '20

Hi, I am currently facing the problem that our Ceph Cluster running Nautilus is only listening on msgrv2 and we are not sure why. This stops us from using block devices via rbd or mounting ceph via the kernel module. Attached[0] you can find the output of 'cat /etc/ceph/ceph.conf', 'ceph mon dump' and 'ceph config dump'. I already asked on IRC and was told that I probably have more success on the mailing list so hopefully someone here also encountered that issue and can help us out. Kind regards, Julian Fölsch -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Stellvertretender Schatzmeister Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

2 2

NFS Ganesha 2.7 in Xenial not available
by Victoria Martinez de la Cruz 23 Jun '20

23 Jun '20

Hi folks, I'm hitting issues with the nfs-ganesha-stable packages [0], the repo url [1] is broken. Is there a known issue for this? Thanks, Victoria [0] https://shaman.ceph.com/repos/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c… [1] https://chacra.ceph.com/r/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c1bac…

3 2

OSD crash with assertion
by Michael Fladischer 23 Jun '20

23 Jun '20

Hi, a lot of our OSD have crashed a few hours ago because of a failed assertion: /build/ceph-15.2.3/src/osd/ECUtil.h: 34: FAILED ceph_assert(stripe_width % stripe_size == 0) Full output here: https://pastebin.com/D1SXzKsK All OSDs are on bluestore and run 15.2.3. I think I messed up when I tried to change an existing EC profile (using --force) for an active EC pool. I already tried to delete the pool and the EC profile and start the OSDs but they keep crashing with the same assertion. Is there a way to at least find out what the values are for stripe_width and stripe_size? Regards, Michael

3 5

Autoscale recommendtion seems to small + it broke my pool...
by Lindsay Mathieson 22 Jun '20

22 Jun '20

Nautilus 14.2.9, setup using Proxmox. * 5 Hosts * 18 OSDs with a mix of disk sizes (3TB, 1TB, 500GB), all bluestore * Pool size = 3, pg_num = 512 According to: https://docs.ceph.com/docs/nautilus/rados/operations/placement-groups/#pres… With 18 OSD's I should be using pg_num=1024, but I actually have it set to 512. However autoscale is recommending pg_num=128 Additionally, I accidentally set autoscale to on, rather than warn, so it started the process. I rapidly got a "Reduced data availability: 2 pgs inactive" warning and io on the pool stopped. I cleared the warning by restarting the effected OSD's for the pg id, but then more cropped up. I only made it stop and restored access to the pool by turning off autoscale and setting pg_num back to 512. Autoscale warning: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE ceph 3239G 3.0 31205G 0.3114 1.0 512 128 warn osd tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 30.47374 - 30 TiB 7.4 TiB 7.4 TiB 28 MiB 23 GiB 23 TiB 24.40 1.00 - root default -5 5.45798 - 5.5 TiB 1.4 TiB 1.3 TiB 4.4 MiB 3.4 GiB 4.1 TiB 24.78 1.02 - host loc 1 hdd 2.72899 0.95001 2.7 TiB 679 GiB 677 GiB 3.1 MiB 1.9 GiB 2.1 TiB 24.31 1.00 136 up osd.1 8 hdd 2.72899 1.00000 2.7 TiB 706 GiB 704 GiB 1.3 MiB 1.5 GiB 2.0 TiB 25.26 1.04 143 up osd.8 -9 1.36449 - 1.4 TiB 431 GiB 429 GiB 1.4 MiB 2.1 GiB 966 GiB 30.84 1.26 - host lod 5 hdd 0.90970 1.00000 932 GiB 291 GiB 290 GiB 1.1 MiB 1.1 GiB 641 GiB 31.20 1.28 59 up osd.5 12 hdd 0.45479 1.00000 466 GiB 140 GiB 139 GiB 293 KiB 1024 MiB 325 GiB 30.12 1.23 28 up osd.12 -11 10.91595 - 11 TiB 2.4 TiB 2.4 TiB 6.9 MiB 5.8 GiB 8.5 TiB 21.94 0.90 - host vnb 6 hdd 2.72899 1.00000 2.7 TiB 613 GiB 612 GiB 2.5 MiB 1.5 GiB 2.1 TiB 21.95 0.90 124 up osd.6 7 hdd 2.72899 1.00000 2.7 TiB 614 GiB 613 GiB 1.7 MiB 1.4 GiB 2.1 TiB 21.98 0.90 124 up osd.7 9 hdd 2.72899 1.00000 2.7 TiB 617 GiB 615 GiB 1.5 MiB 1.4 GiB 2.1 TiB 22.06 0.90 124 up osd.9 17 hdd 2.72899 1.00000 2.7 TiB 608 GiB 607 GiB 1.3 MiB 1.5 GiB 2.1 TiB 21.76 0.89 124 up osd.17 -3 4.54836 - 4.5 TiB 1.3 TiB 1.3 TiB 11 MiB 7.0 GiB 3.3 TiB 27.76 1.14 - host vnh 0 hdd 0.90970 0.95001 932 GiB 220 GiB 219 GiB 3.0 MiB 1021 MiB 711 GiB 23.64 0.97 44 up osd.0 2 hdd 0.90970 0.95001 932 GiB 252 GiB 251 GiB 536 KiB 1023 MiB 679 GiB 27.06 1.11 51 up osd.2 10 hdd 0.54579 1.00000 559 GiB 158 GiB 157 GiB 1.6 MiB 1022 MiB 401 GiB 28.29 1.16 32 up osd.10 11 hdd 0.54579 1.00000 559 GiB 157 GiB 156 GiB 332 KiB 1024 MiB 402 GiB 28.10 1.15 32 up osd.11 14 hdd 0.54579 1.00000 559 GiB 187 GiB 186 GiB 1.1 MiB 1.0 GiB 372 GiB 33.45 1.37 38 up osd.14 15 hdd 0.54579 1.00000 559 GiB 159 GiB 158 GiB 2.0 MiB 1022 MiB 400 GiB 28.51 1.17 32 up osd.15 16 hdd 0.54579 1.00000 559 GiB 159 GiB 158 GiB 2.8 MiB 1021 MiB 400 GiB 28.46 1.17 32 up osd.16 -7 8.18697 - 8.2 TiB 2.0 TiB 2.0 TiB 4.6 MiB 4.5 GiB 6.2 TiB 24.50 1.00 - host vni 3 hdd 2.72899 1.00000 2.7 TiB 670 GiB 669 GiB 1.2 MiB 1.4 GiB 2.1 TiB 23.99 0.98 134 up osd.3 4 hdd 2.72899 1.00000 2.7 TiB 681 GiB 679 GiB 1.6 MiB 1.6 GiB 2.1 TiB 24.36 1.00 136 up osd.4 13 hdd 2.72899 1.00000 2.7 TiB 703 GiB 701 GiB 1.8 MiB 1.5 GiB 2.0 TiB 25.14 1.03 143 up osd.13 TOTAL 30 TiB 7.4 TiB 7.4 TiB 28 MiB 23 GiB 23 TiB 24.40 Should I be reducing the pg_num? is there a way to do it safely? Thanks. -- Lindsay

2 2

radosgw - how to grant read-only access to another user by default
by Paul Choi 22 Jun '20

22 Jun '20

Hi, I'm new to radosgw (learned more about the MDS than I care to...), and it seems like the buckets and objects created by one user cannot be accessed by another user. Is there a way to make any content created by User A accessible (read-only) by User B? From the documentation it looks like this is handled as an S3 permission but I'm not finding an easy/obvious way to do this. Any help would be appreciated. Thanks in advance!

3 2

Re-run ansible to add monitor and RGWs
by Khodayar Doustar 22 Jun '20

22 Jun '20

Hi, I've installed my ceph cluster with ceph-ansible a few months ago. I've just added one monitor and one rgw at that time. So I have 3 nodes, from which one is monitor and rgw and two others only OSD. Now I want to add the other two nodes as monitor and rgw. Can I just modify the ansible host file and re-run the site.yml? I've done some modification in Storage classes, I've added some OSD and uploaded a lot of data up to now. Is it safe to re-run ansible site.yml playbook? I don't want to end with a fresh new cluster! :D Thanks a lot, Khodayar

3 4

nautilus 14.2.9 cluster no bucket auto sharding
by Marcel Kuiper 22 Jun '20

22 Jun '20

I have 3 ceph clusters on nautilus 14.2.9 (same configuration through puppet). 2 of them are autmatically sharding rados buckets, one of them is not When I do radosgw-admin reshard stale-instances list on the cluster where it does not work I get: reshard stale-instances list Resharding disabled in a multisite env, stale instances unlikely from resharding These instances may not be safe to delete. Use --yes-i-really-mean-it to force displaying these instances. The other 2 clusters don't give this warning. They are all single site. The out put of realm list, zonegroup list and zone list for the cluster that fails auto sharding are as follows realm list { "default_info": "e724bd71-31eb-45c8-a456-151f6a5aa8b5", "realms": [ "backup" ] } zonegroup list { "default_info": "ce4329ae-2bc8-4117-9b82-271022b223fa", "zonegroups": [ "dc3" ] } zone list { "default_info": "7f9bebd6-a9cf-4006-83b1-ff99391aacc0", "zones": [ "dc3-r1" ] } mgr configuration (default value) rgw_dynamic_resharding true How does ceph determine whether it is single site or multi site? How can I force the automated bucket sharding?? Any help would be much appreciated Marcel

1 0

Is there a way froce sync metadata in a multisite cluster
by 黄明友 22 Jun '20

22 Jun '20

Hi,all: the slave zone show metadata is caught up with master ; but use radosgw-admin bucket list|wc diff master and the slave zone , is not equal. how can I force sync it?

3 3

Ceph and linux multi queue block IO layer
by Bobby 22 Jun '20

22 Jun '20

Hi all, One question please. Does Ceph uses the linux multi queue block IO layer ? BR Bobby

1 0

RGW slowdown over time
by Glen Baars 22 Jun '20

22 Jun '20

Hello Ceph users, We are experiencing an issue with ceph 14.2.9 / RGW Beast frontend. We are seeing this across our two separate clusters. Over a few weeks the qlen and qactive are going up and not returning to zero. At some point we start seeing performance degrade and we need to reboot the services. We are viewing the queue numbers in the perfcounters_dump. In objecter_requests we aren't seeing any request ( apart from very briefly ) We can reproduce the issue by use S3 browser and setting the concurrent downloads to 100. After completing download of ~1000 files, the queue length has incremented by 2-5 and never returns back to zero. Subsequent bulk downloads increase the qlen. We have the following tunables set rgw_bucket_index_max_aio 128 rgw_dns_name <fqdn##> rgw_frontends beast ssl_port=443 ssl_certificate=<CERT##> rgw_max_chunk_size 4194304 rgw_num_rados_handles 16 rgw_thread_pool_size 500 Anyone seen this or have any idea how to further debug? Any additional tuning suggested? 350TB S3 data Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

1 0

ERROR: osd init failed: (1) Operation not permitted
by Ml Ml 22 Jun '20

22 Jun '20

Hello List, first of all: Yes - i made mistakes. Now i am trying to recover :-/ I had a healthy 3 node cluster which i wanted to convert to a single one. My goal was to reinstall a fresh 3 Node cluster and start with 2 nodes. I was able to healthy turn it from a 3 Node Cluster to a 2 Node cluster. Then the problems began. I started to change size=1 and min_size=1. Health was okay until here. Then over sudden both nodes got fenced...one node refused to boot, mons where missing, etc...to make long story short, here is where i am right now: root@node03:~ # ceph -s cluster b3be313f-d0ef-42d5-80c8-6b41380a47e3 health HEALTH_WARN 53 pgs stale 53 pgs stuck stale monmap e4: 2 mons at {0=10.15.15.3:6789/0,1=10.15.15.2:6789/0} election epoch 298, quorum 0,1 1,0 osdmap e6097: 14 osds: 9 up, 9 in pgmap v93644673: 512 pgs, 1 pools, 1193 GB data, 304 kobjects 1088 GB used, 32277 GB / 33366 GB avail 459 active+clean 53 stale+active+clean root@node03:~ # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 32.56990 root default -2 25.35992 host node03 0 3.57999 osd.0 up 1.00000 1.00000 5 3.62999 osd.5 up 1.00000 1.00000 6 3.62999 osd.6 up 1.00000 1.00000 7 3.62999 osd.7 up 1.00000 1.00000 8 3.62999 osd.8 up 1.00000 1.00000 19 3.62999 osd.19 up 1.00000 1.00000 20 3.62999 osd.20 up 1.00000 1.00000 -3 7.20998 host node02 3 3.62999 osd.3 up 1.00000 1.00000 4 3.57999 osd.4 up 1.00000 1.00000 1 0 osd.1 down 0 1.00000 9 0 osd.9 down 0 1.00000 10 0 osd.10 down 0 1.00000 17 0 osd.17 down 0 1.00000 18 0 osd.18 down 0 1.00000 my main mistakes seemd to be: -------------------------------- ceph osd out osd.1 ceph auth del osd.1 systemctl stop ceph-osd@1 ceph osd rm 1 umount /var/lib/ceph/osd/ceph-1 ceph osd crush remove osd.1 As far as i can tell, ceph waits and needs data from that OSD.1 (which i removed) root@node03:~ # ceph health detail HEALTH_WARN 53 pgs stale; 53 pgs stuck stale pg 0.1a6 is stuck stale for 5086.552795, current state stale+active+clean, last acting [1] pg 0.142 is stuck stale for 5086.552784, current state stale+active+clean, last acting [1] pg 0.1e is stuck stale for 5086.552820, current state stale+active+clean, last acting [1] pg 0.e0 is stuck stale for 5086.552855, current state stale+active+clean, last acting [1] pg 0.1d is stuck stale for 5086.552822, current state stale+active+clean, last acting [1] pg 0.13c is stuck stale for 5086.552791, current state stale+active+clean, last acting [1] [...] SNIP [...] pg 0.e9 is stuck stale for 5086.552955, current state stale+active+clean, last acting [1] pg 0.87 is stuck stale for 5086.552939, current state stale+active+clean, last acting [1] When i try to start ODS.1 manually, i get: -------------------------------------------- 2020-02-10 18:48:26.107444 7f9ce31dd880 0 ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 10210 2020-02-10 18:48:26.134417 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) backend xfs (magic 0x58465342) 2020-02-10 18:48:26.184202 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is supported and appears to work 2020-02-10 18:48:26.184209 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2020-02-10 18:48:26.184526 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2020-02-10 18:48:26.184585 7f9ce31dd880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_feature: extsize is disabled by conf 2020-02-10 18:48:26.309755 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2020-02-10 18:48:26.633926 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.642185 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.664273 7f9ce31dd880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2020-02-10 18:48:26.732154 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for clients 2020-02-10 18:48:26.732163 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400 was 8705, adjusting msgr requires for mons 2020-02-10 18:48:26.732167 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for osds 2020-02-10 18:48:26.732179 7f9ce31dd880 0 osd.1 6002 load_pgs 2020-02-10 18:48:31.939810 7f9ce31dd880 0 osd.1 6002 load_pgs opened 53 pgs 2020-02-10 18:48:31.940546 7f9ce31dd880 -1 osd.1 6002 log_to_monitors {default=true} 2020-02-10 18:48:31.942471 7f9ce31dd880 1 journal close /var/lib/ceph/osd/ceph-1/journal 2020-02-10 18:48:31.969205 7f9ce31dd880 -1 ESC[0;31m ** ERROR: osd init failed: (1) Operation not permittedESC[0m Its mounted: /dev/sdg1 3.7T 127G 3.6T 4% /var/lib/ceph/osd/ceph-1 Is there any way i can get the OSD.1 back in? Thanks a lot, mario

2 1

OSD Keeps crashing, stack trace attached
by Lindsay Mathieson 21 Jun '20

21 Jun '20

I have a problem with one osd (osd.5 on server lod) that keeps crashing. Often it immediately crashes on restart, but oddly a server reboot fixes that, also it alwats starts ok from the command line. Service status and journalctl don't show any useful information. There's two osd's on the server, the other osd never has a problem. Server * osd services only * 8GB Ram * Nautilus 14.2.9 * osd.5 : 1TB - crashes * osd.12 : 500GB - Fine So I ran it from the command line and copied the console dump when it crashed. Any thoughts? should I create a bug report for it? -- Lindsay

1 1

OSD node OS upgrade strategy
by shubjero 21 Jun '20

21 Jun '20

Hi all, I have a 39 node, 1404 spinning disk Ceph Mimic cluster across 6 racks for a total of 9.1PiB raw and about 40% utilized. These storage nodes started their life on Ubuntu 14.04 and in-place upgraded to 16.04 2 years ago however I have started a project to do fresh installs of each OSD node to Ubuntu 18.04 to keep things fresh and well supported. I am reaching out to see what others might suggest in terms of strategy to get these hosts updated quicker than my current strategy. Current strategy: 1. Pick 3 nodes, drain them by modifying the crush weight 2. Fresh install 18.04 using automation tool (MAAS) + some Ansible playbooks to setup server 3. Purge OSD node worth of OSD' (this causes data to be 'misplaced' due to rack weight changing) 4. Run ceph-volume lvm batch for osd node 5. Move OSD's in to desired hosts in crush map (large rebalancing to fill back up) If anyone has suggestions on a quicker way to do this I am all ears. I am wondering if its not necessary to have to drain/fill OSD nodes at all and if this can be done with just a fresh install and not touch the OSD's however I don't know how to perform a fresh installation and then tell ceph that I have OSD's with data on them and to somehow re-register them with the cluster? Or is there a better order of operations to draining/filling without causing a high amount of objects to be misplaced due to manipulating the crush map. That being said, since our cluster is a bit older and the majority of our bluestore osd's are provisioned in the 'simple' method using a small metadata partition and the remainder as a raw partition whereas now it seems the suggested way is to use the lvm layout and tmpfs. Anyways, I'm all ears and appreciate any feedback. Jared Baker Ontario Institute for Cancer Research

3 2

bluestore_rocksdb_options
by Seena Fallah 20 Jun '20

20 Jun '20

Hi. I found a default rocksdb option in bluestore that I can't find in facebook rocksdb. recycle_log_file_num this config if a boolean config in facebook rocksdb but in default Ceph configs the value of this is 4. Can someone tell what it means?

2 6

meta values on nvme class OSDs
by Emre Eryilmaz 20 Jun '20

20 Jun '20

Hi, Ceph version 12.2.13. Ceph status show ‘Health_Warn’ nearfull osd(s) and pool(s). No data in this nvme class pools and nvme osd’s, completely free. But meta values is so big in nvme osds. Why does meta values on OSDs shows huge disk usage? $ ceph health detail HEALTH_WARN 1 nearfull osd(s); 6 pool(s) nearfull OSD_NEARFULL 1 nearfull osd(s) osd.49 is near full POOL_NEARFULL 6 pool(s) nearfull pool 'VmImages_NVMe_FdChassis_Rp3' is nearfull pool '.rgw.root' is nearfull pool 'default.rgw.control' is nearfull pool 'default.rgw.meta' is nearfull pool 'default.rgw.log' is nearfull pool 'default.rgw.buckets.non-ec' is nearfull -- $ ceph df --cluster ceph GLOBAL: SIZE AVAIL RAW USED %RAW USED 501TiB 354TiB 147TiB 29.42 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS VmImages_SSD_FdHost_Rp3 6 25.2TiB 33.26 50.6TiB 6990169 VmImages_NVMe_FdChassis_Rp3 8 19B 0 242GiB 2 VmImages_HDD_FdHost_Rp3 15 21.3TiB 35.33 39.0TiB 5755928 .rgw.root 16 1.77KiB 0 16.3TiB 11 default.rgw.control 17 0B 0 16.3TiB 8 default.rgw.meta 18 17.6KiB 0 16.3TiB 80 default.rgw.log 19 162B 0 16.3TiB 210 default.rgw.buckets.index 20 0B 0 50.6TiB 252 default.rgw.buckets.data 21 95.8GiB 0.24 39.0TiB 1561373 default.rgw.buckets.non-ec 22 0B 0 16.3TiB 4 default.rgw.buckets.data.ssd 25 0B 0 50.6TiB 0 $ ceph osd df tree | grep nvme ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME 0 nvme 0.36378 1.00000 373GiB 313GiB 57.3MiB 25.8MiB 313GiB 59.0GiB 84.15 2.86 79 osd.0 7 nvme 0.36378 1.00000 373GiB 307GiB 57.4MiB 22.9MiB 307GiB 65.8GiB 82.32 2.80 70 osd.7 14 nvme 0.36378 1.00000 373GiB 313GiB 57.3MiB 25.9MiB 313GiB 59.3GiB 84.08 2.86 79 osd.14 21 nvme 0.36378 1.00000 373GiB 312GiB 57.4MiB 24.2MiB 312GiB 60.8GiB 83.68 2.85 78 osd.21 28 nvme 0.36378 1.00000 373GiB 316GiB 57.3MiB 29.8MiB 316GiB 56.4GiB 84.87 2.89 85 osd.28 35 nvme 0.36378 1.00000 373GiB 312GiB 57.4MiB 26.5MiB 312GiB 60.5GiB 83.75 2.85 80 osd.35 42 nvme 0.36378 1.00000 373GiB 311GiB 57.5MiB 25.5MiB 311GiB 61.9GiB 83.39 2.84 79 osd.42 49 nvme 0.36378 1.00000 373GiB 318GiB 57.3MiB 27.7MiB 318GiB 54.8GiB 85.28 2.90 90 osd.49 56 nvme 0.36378 1.00000 373GiB 303GiB 57.3MiB 20.2MiB 303GiB 69.5GiB 81.33 2.77 71 osd.56 63 nvme 0.36378 1.00000 373GiB 309GiB 57.4MiB 28.5MiB 309GiB 63.2GiB 83.02 2.82 82 osd.63 70 nvme 0.36378 1.00000 373GiB 312GiB 57.3MiB 26.2MiB 312GiB 60.5GiB 83.76 2.85 88 osd.70 77 nvme 0.36378 1.00000 373GiB 299GiB 57.4MiB 22.7MiB 299GiB 73.6GiB 80.24 2.73 69 osd.77 84 nvme 0.36378 1.00000 373GiB 312GiB 57.4MiB 31.7MiB 312GiB 60.2GiB 83.84 2.85 76 osd.84 91 nvme 0.36378 1.00000 373GiB 301GiB 57.3MiB 21.1MiB 301GiB 71.5GiB 80.80 2.75 76 osd.91 98 nvme 0.36378 1.00000 373GiB 285GiB 57.3MiB 15.7MiB 285GiB 87.6GiB 76.47 2.60 54 osd.98 105 nvme 0.36378 1.00000 373GiB 308GiB 57.3MiB 30.8MiB 308GiB 64.6GiB 82.67 2.81 86 osd.105 112 nvme 0.36378 1.00000 373GiB 264GiB 57.3MiB 26.0MiB 264GiB 108GiB 70.99 2.42 84 osd.112 119 nvme 0.36378 1.00000 373GiB 297GiB 57.3MiB 23.6MiB 297GiB 75.1GiB 79.85 2.72 74 osd.119 126 nvme 0.36378 1.00000 373GiB 292GiB 57.3MiB 22.8MiB 292GiB 80.2GiB 78.48 2.67 68 osd.126 133 nvme 0.36378 1.00000 373GiB 294GiB 57.3MiB 24.1MiB 294GiB 78.7GiB 78.87 2.68 70 osd.133

1 0

meta values on nvme class OSDs
by Emre Eryilmaz 20 Jun '20

20 Jun '20

Hi, Ceph version 12.2.13. Ceph status show ‘Health_Warn’ nearfull osd(s) and pool(s). No data in this nvme class pools and nvme osd’s, completely free. But meta values is so big in nvme osds. Why does meta values on OSDs shows huge disk usage? $ ceph health detail HEALTH_WARN 1 nearfull osd(s); 6 pool(s) nearfull OSD_NEARFULL 1 nearfull osd(s) osd.49 is near full POOL_NEARFULL 6 pool(s) nearfull pool 'VmImages_NVMe_FdChassis_Rp3' is nearfull pool '.rgw.root' is nearfull pool 'default.rgw.control' is nearfull pool 'default.rgw.meta' is nearfull pool 'default.rgw.log' is nearfull pool 'default.rgw.buckets.non-ec' is nearfull -- $ ceph df --cluster ceph GLOBAL: SIZE AVAIL RAW USED %RAW USED 501TiB 354TiB 147TiB 29.42 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS VmImages_SSD_FdHost_Rp3 6 25.2TiB 33.26 50.6TiB 6990169 VmImages_NVMe_FdChassis_Rp3 8 19B 0 242GiB 2 VmImages_HDD_FdHost_Rp3 15 21.3TiB 35.33 39.0TiB 5755928 .rgw.root 16 1.77KiB 0 16.3TiB 11 default.rgw.control 17 0B 0 16.3TiB 8 default.rgw.meta 18 17.6KiB 0 16.3TiB 80 default.rgw.log 19 162B 0 16.3TiB 210 default.rgw.buckets.index 20 0B 0 50.6TiB 252 default.rgw.buckets.data 21 95.8GiB 0.24 39.0TiB 1561373 default.rgw.buckets.non-ec 22 0B 0 16.3TiB 4 default.rgw.buckets.data.ssd 25 0B 0 50.6TiB 0 $ ceph osd df tree | grep nvme ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME 0 nvme 0.36378 1.00000 373GiB 313GiB 57.3MiB 25.8MiB 313GiB 59.0GiB 84.15 2.86 79 osd.0 7 nvme 0.36378 1.00000 373GiB 307GiB 57.4MiB 22.9MiB 307GiB 65.8GiB 82.32 2.80 70 osd.7 14 nvme 0.36378 1.00000 373GiB 313GiB 57.3MiB 25.9MiB 313GiB 59.3GiB 84.08 2.86 79 osd.14 21 nvme 0.36378 1.00000 373GiB 312GiB 57.4MiB 24.2MiB 312GiB 60.8GiB 83.68 2.85 78 osd.21 28 nvme 0.36378 1.00000 373GiB 316GiB 57.3MiB 29.8MiB 316GiB 56.4GiB 84.87 2.89 85 osd.28 35 nvme 0.36378 1.00000 373GiB 312GiB 57.4MiB 26.5MiB 312GiB 60.5GiB 83.75 2.85 80 osd.35 42 nvme 0.36378 1.00000 373GiB 311GiB 57.5MiB 25.5MiB 311GiB 61.9GiB 83.39 2.84 79 osd.42 49 nvme 0.36378 1.00000 373GiB 318GiB 57.3MiB 27.7MiB 318GiB 54.8GiB 85.28 2.90 90 osd.49 56 nvme 0.36378 1.00000 373GiB 303GiB 57.3MiB 20.2MiB 303GiB 69.5GiB 81.33 2.77 71 osd.56 63 nvme 0.36378 1.00000 373GiB 309GiB 57.4MiB 28.5MiB 309GiB 63.2GiB 83.02 2.82 82 osd.63 70 nvme 0.36378 1.00000 373GiB 312GiB 57.3MiB 26.2MiB 312GiB 60.5GiB 83.76 2.85 88 osd.70 77 nvme 0.36378 1.00000 373GiB 299GiB 57.4MiB 22.7MiB 299GiB 73.6GiB 80.24 2.73 69 osd.77 84 nvme 0.36378 1.00000 373GiB 312GiB 57.4MiB 31.7MiB 312GiB 60.2GiB 83.84 2.85 76 osd.84 91 nvme 0.36378 1.00000 373GiB 301GiB 57.3MiB 21.1MiB 301GiB 71.5GiB 80.80 2.75 76 osd.91 98 nvme 0.36378 1.00000 373GiB 285GiB 57.3MiB 15.7MiB 285GiB 87.6GiB 76.47 2.60 54 osd.98 105 nvme 0.36378 1.00000 373GiB 308GiB 57.3MiB 30.8MiB 308GiB 64.6GiB 82.67 2.81 86 osd.105 112 nvme 0.36378 1.00000 373GiB 264GiB 57.3MiB 26.0MiB 264GiB 108GiB 70.99 2.42 84 osd.112 119 nvme 0.36378 1.00000 373GiB 297GiB 57.3MiB 23.6MiB 297GiB 75.1GiB 79.85 2.72 74 osd.119 126 nvme 0.36378 1.00000 373GiB 292GiB 57.3MiB 22.8MiB 292GiB 80.2GiB 78.48 2.67 68 osd.126 133 nvme 0.36378 1.00000 373GiB 294GiB 57.3MiB 24.1MiB 294GiB 78.7GiB 78.87 2.68 70 osd.133

1 0

Mapped RBD is too slow?
by Michal.Plsek＠seznam.cz 19 Jun '20

19 Jun '20

Good day, rw operations (randwrite 4kB and 4MB) over mapped RBD are just too slow. I am also using librbd over TGT. fio input: [global] rw=randwrite ioengine=libaio iodepth=64 size=1g direct=1 buffered=0 startdelay=5 group_reporting=1 thread=1 ramp_time=5 time_based disk_util=0 clat_percentiles=0 disable_lat=1 disable_clat=1 disable_slat=1 #numjobs=16 runtime=60 filename=/mnt/disk/test1/testfile.fio [test] name=test bs=4k stonewall fio output for TGT (librbd): test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- 4096B, ioengine=libaio, iodepth=64 Starting 1 thread test: Laying out IO files (2 files / total 1024MiB) test: (groupid=0, jobs=1): err= 0: pid=6909: Fri Jun 19 15:26:11 2020 write: IOPS=6342, BW=24.8MiB/s (25.0MB/s)(1487MiB/60003msec) bw ( KiB/s): min= 8, max=70216, per=100.00%, avg=30441.30, stdev= 28899.02, samples=100 iops : min= 2, max=17554, avg=7610.27, stdev=7224.76, samples= 100 cpu : usr=2.18%, sys=11.08%, ctx=107852, majf=0, minf=356 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64= 115.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64= 0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64= 0.0% issued rwts: total=0,380583,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=24.8MiB/s (25.0MB/s), 24.8MiB/s-24.8MiB/s (25.0MB/s-25.0MB/s), io=1487MiB (1559MB), run=60003-60003msec ----------------- fio output for RBD: test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- 4096B, ioengine=libaio, iodepth=64 Starting 1 thread test: (groupid=0, jobs=1): err= 0: pid=7372: Fri Jun 19 15:33:51 2020 write: IOPS=909, BW=3642KiB/s (3729kB/s)(214MiB/60186msec) bw ( KiB/s): min= 2792, max= 4688, per=100.00%, avg=3648.13, stdev= 399.09, samples=120 iops : min= 698, max= 1172, avg=912.01, stdev=99.75, samples=120 cpu : usr=0.78%, sys=3.08%, ctx=37108, majf=0, minf=267 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64= 110.2% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64= 0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64= 0.0% issued rwts: total=0,54732,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=3642KiB/s (3729kB/s), 3642KiB/s-3642KiB/s (3729kB/s-3729kB/s), io=214MiB (224MB), run=60186-60186msec ----------------- I ran these tests from separate client server. I suspect that RBD is not working correctly in there, since I tried some fio tests on it and the result was almost the same with RBD cache set to false in ceph.conf (for example: fio -ioengine=libaio -name=test -bs=4M -iodepth=64 -numjobs=16 -rw= randwrite -direct=1 -runtime=60 -filename=/mnt/disk/test1 -size=10g). Can you give me any ideas where the problem might be, perhaps with RBD cache? Network capacity and usual things has been tested already. I will be able to provide more specs if needed. Thanks!

1 0

Radosgw huge traffic to index bucket compared to incoming requests
by Mariusz Gronczewski 19 Jun '20

19 Jun '20

Hi, we're using Ceph as S3-compatible storage to serve static files (mostly css/js/images + some videos) and I've noticed that there seem to be huge read amplification for index pool. Incoming traffic magniture is of around 15k req/sec (mostly sub 1MB request but index pool is getting hammered: pool pl-war1.rgw.buckets.index id 10 client io 632 MiB/s rd, 277 KiB/s wr, 129.92k op/s rd, 415 op/s wr pool pl-war1.rgw.buckets.data id 11 client io 4.5 MiB/s rd, 6.8 MiB/s wr, 640 op/s rd, 1.65k op/s wr and is getting order of magnitude more requests running 15.2.3, nothing special in terms of tunning aside from disabling some logging as to not overflow the logs. We've had similar test cluster on 12.x (and way slower hardware) getting similar traffic and haven't observed that magnitude of difference. when enabling debug on affected OSD I only get spam of 2020-06-17T12:35:05.700+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) omap_get_header 10.51_head oid #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# = 0 2020-06-17T12:35:05.700+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.700+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.700+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.700+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b1a34d8:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.214:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) omap_get_header 10.51_head oid #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# = 0 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.704+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.708+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.708+0200 7f80694c4700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b0d75b0:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.222:head# 2020-06-17T12:35:05.716+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) omap_get_header 10.51_head oid #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# = 0 2020-06-17T12:35:05.716+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# 2020-06-17T12:35:05.716+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# 2020-06-17T12:35:05.720+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# 2020-06-17T12:35:05.720+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# 2020-06-17T12:35:05.720+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# 2020-06-17T12:35:05.720+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# 2020-06-17T12:35:05.720+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# 2020-06-17T12:35:05.720+0200 7f806d4cc700 10 bluestore(/var/lib/ceph/osd/ceph-20) get_omap_iterator 10.51_head #10:8b5ed205:::.dir.88d4f221-0da5-444d-81a8-517771278350.454759.8.151:head# -- Mariusz Gronczewski (XANi) <xani666(a)gmail.com> GnuPG: 0xEA8ACE64 http://devrandom.pl

3 5

Orchestrator: Cannot add node after mistake
by Simon Sutter 19 Jun '20

19 Jun '20

Hello, I did a mistake, while deploying a new node on octopus. The node is a fresh installed Centos8 machine. Bevore I did a "ceph orch host add node08" I pasted the wrong command: ceph orch daemon add osd node08:cl_node08/ceph That did not return anything, so I tried to add the node first with the host add command, but now I get an error: Error ENOENT: New host node08 (node08) failed check: ['Traceback (most recent call last):', ' File "<stdin>", line 4580, in <module>', ' File "<stdin>", line 3592, in command_check_host', "UnboundLocalError: local variable 'container_path' referenced before assignment"] I'm not a developer, so I don't know where to look, and how to fix this. I tried to reboot every node, to see if it's just a cached problem, but no luck there. Do any of you know how to fix this? Thanks in advance, Simon

2 3

Enable msgr2 mon service restarted
by Amit Ghadge 18 Jun '20

18 Jun '20

Hello All, We saw all nodes mon services are restarted at the same time after enabling msgr2, So this make an impact on production running cluster? We are upgrading from Luminous to Nautilus. Thanks, AmitG

1 1

Can't bind mon to v1 port in Octopus.
by Miguel Afonso 18 Jun '20

18 Jun '20

Hi, I have a lab single node cluster with octopus installed via ceph-ansible. Both v1 and v2 were enabled in ceph-ansible vars with the correct suffixes. The configuration was generated correctly and both ports were included in the mon array. [global] cluster network = 172.16.6.0/24 fsid = bb204a5c-957d-4a06-a372-redacted mon_host = [v2:172.16.6.210:3300/0,v1:172.16.6.210:6789/0] mon initial members = aio1 mon_pg_warn_max_per_osd = 0 osd pool default crush rule = -1 osd_pool_default_min_size = 1 osd_pool_default_size = 1 public network = 172.16.6.0/24 I can also see that `ms_bind_msgr1` is enabled in the live config. root@aio1 ~ # ceph daemon mon.aio1 config show | grep msgr "mon_warn_on_msgr2_not_enabled": "true", "ms_bind_msgr1": "true", "ms_bind_msgr2": "true", However only v2 is binding netstat -tlnp |grep mon tcp 0 0 172.16.6.210:3300 0.0.0.0:* LISTEN 2039098/ceph-mon I have a client that only speaks v1 (ceph-csi) that can't talk to the v2 port 2020-06-15T09:49:51.330+0100 7f8776038700 -1 --2- v2:172.16.6.210:3300/0 >> conn(0x563bfd6b2000 0x563bde5ff600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer is using msgr V1 protocol 2020-06-15T09:49:52.258+0100 7f8776038700 -1 --2- v2:172.16.6.210:3300/0 >> conn(0x563bfd6b2000 0x563bde5ff600 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer is using msgr V1 protocol What could be the reason for mon not binding to port 6789? Thanks Miguel

3 3

Re: ceph grafana dashboards: rbd overview empty
by Zhenshi Zhou 18 Jun '20

18 Jun '20

Yep, you should also tell mgr that rbd of which pool you wanna export statistics. Follow this, https://ceph.io/rbd/new-in-nautilus-rbd-performance-monitoring/ Marc Roos <M.Roos(a)f1-outsourcing.eu> 于2020年6月12日周五下午10:33写道： > > The grafana dashboard 'rbd overview' is empty. Queries have measurements > 'ceph_rbd_write_ops' that do not exist in prometheus (I think). Should I > enable something more than just 'ceph mgr module enable prometheus' > > I am on Nautilus > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

ceph mds slow requests
by locallocal 18 Jun '20

18 Jun '20

hi,guys. we have a ceph cluster which version is luminous 12.2.13. and Recently we encountered a problem.here are some log infomations: 2020-06-08 12:33:52.706070 7f4097e2d700 0 log_channel(cluster) log [WRN] : slow request 30.518930 seconds old, received at 2020-06-08 12:33:22.186924: client_request(client.48978906:941633993 create #0x100028cab8a/.filename 2020-06-08 12:33:22.197434 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply ... 2020-06-08 13:12:17.826727 7f4097e2d700 0 log_channel(cluster) log [WRN] : slow request 2220.991833 seconds old, received at 2020-06-08 12:35:16.764233: client_request(client.42390705:788369155 create #0x1000224f999/.filename 2020-06-08 12:35:16.774553 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply it looks like mds can't flush journal to osd of meta pool.but the osd type is ssd and the load is very low.this problem leads the client can't mount and the mds can't trim log. Is there anyone have encountered this problem.Please help!

4 7

Jewel clients on recent cluster
by Christoph Ackermann 18 Jun '20

18 Jun '20

Hi all, we have a cluster starting from jewel to octopus nowadays. We would like to enable Upmap but unfortunately there are some old Jewel clients active. We cannot force Upmap by: ceph osd set-require-min-compat-client luminous Because of production state, we must not lose any client. ;-) "client": [ { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 7 }, { "features": "0x3f01cfb8ffadffff", "release": "luminous", "num": 6 } The cluster and all clients are v15.2.3 and my assumtion was, that Centos7 with kernel 3.10 have backported kernel modules. Am i wrong? I also checked Centos7 client with 4.20-ml kernel without success. Clients always appear as Jewel clients... Fresh Centos8 run as Luminous client as expected. BTW: Is there a trick to identify Jewel clients by IP address / Hostname? Thank you much, Christoph -- -- Christoph Ackermann | System Engineer INFOSERVE GmbH | Am Felsbrunnen 15 | D-66119 Saarbrücken Fon +49 (0)681 88008-59 | Fax +49 (0)681 88008-33 | mailto:C.Ackermann@infoserve.de | https://www.infoserve.de INFOSERVE Datenschutzhinweise: https://infoserve.de/datenschutz Handelsregister: Amtsgericht Saarbrücken, HRB 11001 | Erfüllungsort: Saarbrücken Geschäftsführer: Dr. Stefan Leinenbach | Ust-IdNr.: DE168970599

3 3

OSD heartbeat failure
by neil.ashby-senior＠bt.com 18 Jun '20

18 Jun '20

Hi, I have a Luminous (12.2.25) cluster with several OSDs down. The daemons start but they're reporting as down. I did see in some osd logs that heartbeats were failing but when I checked the ports for the heartbeats were incorrect for that osd, although another osd was listening on that. How does the osd know what ports to ping other osds on? Is there any way to force an update. The reason this happened is because someone took a VM snapshot of this cluster and restored the snapshot so the osds aren't up. I know this isn't a good implementation or a good idea and this will change going forward. Anyway, I was just wondering about the heartbeat issue and whether attempting to ping on the right ports might bring them up. Thanks, Neil.

2 1

How to force backfill on undersized pgs ?
by Kári Bertilsson 18 Jun '20

18 Jun '20

Hello I'm running ceph 14.2.9. During heavy backfilling due to rebalancing one OSD crashed. I want to recover the data from the lost OSD before continuing the backfilling so i out'ed the lost osd and ran "ceph osd set norebalance". But i'm noticing with the norebalance flag set the system does not backfill the undersized PG's. Only the degraded ones. So now i have plenty of undersized PG's and the system is idle. How can i recover the undersized PG's before resuming normal backfilling/rebalancing ? Regards, Kári

2 1