July 2020 - ceph-users - lists.ceph.io

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

v15.2.4 Octopus released
by David Galloway 17 Dec '20

17 Dec '20

We're happy to announce the fourth bugfix release in the Octopus series. In addition to a security fix in RGW, this release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-4-octopus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * Cephadm: There were a lot of small usability improvements and bug fixes: * Grafana when deployed by Cephadm now binds to all network interfaces. * `cephadm check-host` now prints all detected problems at once. * Cephadm now calls `ceph dashboard set-grafana-api-ssl-verify false` when generating an SSL certificate for Grafana. * The Alertmanager is now correctly pointed to the Ceph Dashboard * `cephadm adopt` now supports adopting an Alertmanager * `ceph orch ps` now supports filtering by service name * `ceph orch host ls` now marks hosts as offline, if they are not accessible. * Cephadm can now deploy NFS Ganesha services. For example, to deploy NFS with a service id of mynfs, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:: ceph orch apply nfs mynfs nfs-ganesha nfs-ns * Cephadm: `ceph orch ls --export` now returns all service specifications in yaml representation that is consumable by `ceph orch apply`. In addition, the commands `orch ps` and `orch ls` now support `--format yaml` and `--format json-pretty`. * Cephadm: `ceph orch apply osd` supports a `--preview` flag that prints a preview of the OSD specification before deploying OSDs. This makes it possible to verify that the specification is correct, before applying it. * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, and `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * RBD: The name of the rbd pool object that is used to store rbd trash purge schedule is changed from "rbd_trash_trash_purge_schedule" to "rbd_trash_purge_schedule". Users that have already started using `rbd trash purge schedule` functionality and have per pool or namespace schedules configured should copy "rbd_trash_trash_purge_schedule" object to "rbd_trash_purge_schedule" before the upgrade and remove "rbd_trash_purge_schedule" using the following commands in every RBD pool and namespace where a trash purge schedule was previously configured:: rados -p <pool-name> [-N namespace] cp rbd_trash_trash_purge_schedule rbd_trash_purge_schedule rados -p <pool-name> [-N namespace] rm rbd_trash_trash_purge_schedule or use any other convenient way to restore the schedule after the upgrade. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7447c15c6ff58d7fce91843b705a268a1917325c -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

4 5

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Advice on SSD choices for WAL/DB?
by Andrei Mikhailovsky 26 Nov '20

26 Nov '20

Hello, We are planning to perform a small upgrade to our cluster and slowly start adding 12TB SATA HDD drives. We need to accommodate for additional SSD WAL/DB requirements as well. Currently we are considering the following: HDD Drives - Seagate EXOS 12TB SSD Drives for WAL/DB - Intel D3 S4510 960GB or Intel D3 S4610 960GB Our cluster isn't hosting any IO intensive DBs nor IO hungry VMs such as Exchange, MSSQL, etc. From the documentation that I've read the recommended size for DB is between 1% and 4% of the size of the osd. Would 2% figure be sufficient enough (so around 240GB DB size for each 12TB osd?) Also, from your experience, which is a better model for the SSD DB/WAL? Would Intel S4510 be sufficient enough for our purpose or would the S4610 be a much better choice? Are there any other cost effective performance to consider instead of the above models? The same question to the HDD. Any other drives we should consider instead of the Seagate EXOS series? Thanks for you help and suggestions. Andrei

4 4

OSD memory leak?
by Frank Schilder 16 Nov '20

16 Nov '20

Hi all, on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load. What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats: Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } } Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Am I looking at a memory leak here or are these heap stats expected? I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 25

NoSuchKey on key that is visible in s3 list/radosgw bk
by Mariusz Gronczewski 06 Nov '20

06 Nov '20

Hi, I've got a problem on Octopus (15.2.3, debian packages) install, bucket S3 index shows a file: s3cmd ls s3://upvid/255/38355 --recursive 2020-07-27 17:48 50584342 s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 radosgw-admin bi list also shows it { "type": "plain", "idx": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "entry": { "name": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "instance": "", "ver": { "pool": 11, "epoch": 853842 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 50584342, "mtime": "2020-07-27T17:48:27.203008Z", "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", "storage_class": "", "owner": "filmweb-app", "owner_display_name": "filmweb app user", "content_type": "", "accounted_size": 50584342, "user_data": "", "appendable": "false" }, "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, but trying to download it via curl (I've set permissions to public0 only gets me <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>upvid</BucketName><RequestId>tx0000000000000000e716d-005f1f14cb-e478a-pl-war1</RequestId><HostId>e478a-pl-war1-pl</HostId></Error> (the actually nonexisting files shows access denied in same context) same with other tools: $ s3cmd get s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 /tmp download: 's3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' -> '/tmp/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' [1 of 1] ERROR: S3 error: 404 (NoSuchKey) cluster health is OK Any ideas what is happening here ? -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: admin(a)efigence.com

3 3

Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early
by Wido den Hollander 28 Sep '20

28 Sep '20

Hi, On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing OSDs randomly drop out of the cluster. Usually it's 2 to 4 OSDs spread out over different nodes. Each node has 16 OSDs and not all the failing OSDs are on the same node. The OSDs are marked as down and all they keep print in their logs: monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-06-04T07:57:17.706529-0400) Looking at their status through the admin socket: { "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", "whoami": 206, "state": "active", "oldest_map": 73697, "newest_map": 75795, "num_pgs": 19 } The message brought me to my own ticket I created 2 years ago: https://tracker.ceph.com/issues/23460 The first thing I've checked is NTP/time. Double, triple check this. All the times are in sync on the cluster. Nothing wrong there. Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. Restarting them brings them back right away and then within 24h some other OSDs will drop out. Has anybody seen this behavior with Octopus as well? Wido

2 1

Solve the issue of time scarcity via assignment help in Kuwait
by kevin wick 24 Sep '20

24 Sep '20

Are you facing a scarcity of timings or lack of time while composing assignments for Kuwait Universities? Are you not sure while taking the help of anybody to compose your assignments? For every student, assignments sound necessary but time-consuming tasks. You have to manage your time for your projects to score high marks as you can’t ignore your assignments during your study tenure. In this situation, if you can’t manage your time and require reliable assistance for your important task, place your order for assignment help even in Kuwait. Two important things that you need to keep in mind while working on your assignments are time management and quality content. Most students fail to submit their assignments on time because they could not manage their time and can’t collect relevant information for drafting their academic papers. However, don’t burst out and suffer your marks because of any reason. Instead of it, connect with assignment writing service provider and get your work done on time if you have less time to write your assignment. Many students achieve high marks on their projects due to the assistance of experts and professional writers. When you quote order for assignment writing help, you will provide enough time to engage yourself in some other academic tasks. If you don’t have time for writing your assignments and have no idea how to collect particulars for your work, transfer your project to professionals. Experts know how to arrange relevant information for composing the effective academic papers so you will not lose your marks. Expert’s knowledge and experience will allow you to connect with the right source of information and help you to score high marks. So, if you have issues in writing your academic papers, don’t forget to check out the services of online academic writing. https://www.greatassignmenthelp.com/kw/

3 2

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

cephadm - How to deploy ceph cluster with a partition on SSD for block.db
by klemen＠psi-net.si 11 Sep '20

11 Sep '20

I'm trying to deploy a ceph cluster with a cephadm tool. I've already successfully done all steps except adding OSDs. My testing equipment consists of three hosts. Each host has SSD storage, where OS is installed into. On that storage I created partition, which can be used as a ceph block.db. Hosts have also 2 additional HDs (spinning drives) for OSD data. On docs I couldn't find how to deploy such configuration. Do you have any hints, how to do that? Thanks for help!

5 7

Release mental stress using Hong Kong Assignment help
by james wick 08 Sep '20

08 Sep '20

Many students can’t complete their assignments within the assigned date because of some unavoidable circumstances in Hong Kong. It’s true that your mental stress raises hindrance and disturbs your concentration for your work. Because of stress and tension, you will not have enough thoughts to make your work effective. This situation directs you to the platform of Assignment Help services even in Hong Kong. This is because an unstable mind could not generate the right ideas to compose a worthy assignment. Make your assignment informative and productive using the online writing services of assignment experts. Professionals have a better understanding of the subject and know how to frame all information in the right format. So, you can use online assignment help when you have issues in composing your academic papers irrespective of any subject. https://www.greatassignmenthelp.com/hk/

4 3

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

[Ceph Octopus 15.2.3 ] MDS crashed suddenly
by carlimeunier＠gmail.com 27 Aug '20

27 Aug '20

Hi, I made a fresh install of Ceph Octopus 15.2.3 recently. And after a few days, the 2 standby MDS suddenly crashed with segmentation fault error. I try to restart it but it does not start. Here is the error : -20> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _renew_subs -19> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _send_mon_message to mon.2 at v1:172.31.36.98:6789/0 -18> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcf9530c0 version 269 -17> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa87520 version 269 -16> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa875c0 version 269 -15> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa871c0 version 269 -14> 2020-07-17T13:50:27.888+0000 7fc8c8c55700 10 monclient: get_auth_request con 0x559dcfada000 auth_method 0 -13> 2020-07-17T13:50:27.888+0000 7fc8c9456700 10 monclient: get_auth_request con 0x559dcfada800 auth_method 0 -12> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) recover start -11> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) read_head -10> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 4 mds.0.log Waiting for journal 0x200 to recover... -9> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_read_head loghead(trim 4194304, expire 4231216, write 4329405, stream_format 1). probing for end of log (from 4329405)... -8> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) probing for end of the log -7> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_probe_end write_pos = 4329949 (header had 4329405). recovered. -6> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Journal 0x200 recovered. -5> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Recovered journal 0x200 in format 1 -4> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 2 mds.0.0 Booting: 1: loading/discovering base inodes -3> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x100 -2> 2020-07-17T13:50:27.894+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x1 -1> 2020-07-17T13:50:27.894+0000 7fc8c0444700 2 mds.0.0 Booting: 2: replaying mds log 0> 2020-07-17T13:50:27.896+0000 7fc8bec41700 -1 *** Caught signal (Segmentation fault) ** in thread 7fc8bec41700 thread_name:md_log_replay Here is the cluster information : # ceph status cluster: id: dd024fe1-4996-4fed-ba57-03090e53724d health: HEALTH_WARN 20 daemons have recently crashed services: mon: 3 daemons, quorum 2,0,1 (age 2d) mgr: mgr.0(active, since 9d), standbys: mgr.2, mgr.1 mds: cephfs:1 {0=node0=up:active} 1 up:standby-replay 1 up:standby osd: 3 osds: 3 up (since 28h), 3 in (since 9d) task status: scrub status: mds.node0: idle mds.node2: idle data: pools: 3 pools, 49 pgs objects: 29 objects, 170 KiB usage: 3.0 GiB used, 41 TiB / 41 TiB avail pgs: 49 active+clean io: client: 853 B/s rd, 1 op/s rd, 0 op/s wr There is only 1 client connected to the cluster. Please, does anyone have any idea? Thanks

3 4

kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay)
by Jake Grimmett 27 Aug '20

27 Aug '20

Dear all, After enabling "allow_standby_replay" on our cluster we are getting (lots) of identical errors on the client /var/log/messages like Apr 29 14:21:26 hal kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay) We are using the ml kernel 5.6.4-1.el7 on Scientific Linux 7.8 Cluster and client are running Ceph v14.2.9 Setting was enabled with: # ceph fs set cephfs allow_standby_replay true [root@ceph-s1 ~]# ceph mds stat cephfs:1 {0=ceph-s3=up:active} 1 up:standby-replay 2 up:standby Is this something to worry about, or should we just disable allow_standby_replay ? any advice appreciated, many thanks Jake Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

2 2

OSDs get full with bluestore logs
by Khodayar Doustar 21 Aug '20

21 Aug '20

Hi, I have a 3 node cluster of mimic with 9 osds (3 osds on each node). I use this cluster to test integration of an application with S3 api. The problem is that after a few days all OSD starts filling up with bluestore logs and goes down and out one by one! I cannot stop the logs and I cannot find the setting to fix this leakage, this should be a leakage in logs because it's not logical to fill up all OSD with bluefs logs. This is an example of logs which is being repeated in bluestore logs: [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path /var/lib/ceph/osd/ceph-5 . . . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 15:54:29.056488 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 15:54:30.057828 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 15:54:31.059252 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0xf5f0) [0x7f40fd47f5f0] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] 4: (BlueFS::log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x3b) [0x55603a1fad3b] 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x64) [0x55603a1da764] 6: (main()+0x2f3e) [0x55603a105e3e] 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] 8: (()+0x23115f) [0x55603a1d915f] 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- . . . And this is the output of daemonperf for one of remaining (filling up!) OSDs: [root@server2 ~]# ceph daemonperf osd.5 ------bluefs------- ------------bluestore------------- ----------osd----------- jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l rop | 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 ^C[root@server2 ~]# And this is the enospc logs when trying to start OSD: [root@server2 ~]# journalctl -u -f ceph-osd(a)4.service Failed to add match 'ceph-osd(a)4.service': Invalid argument Failed to add filters: Invalid argument [root@server2 ~]# journalctl -f -u ceph-osd(a)4.service -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10] Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon osd.4. Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon osd.4. and some more data: [root@server1 ~]# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 38 GiB 35 GiB 2.6 GiB 6.83 303 POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED .rgw.root 18 N/A N/A 1.1 KiB 0 50 GiB 4 4 63 B 4 B 1.1 KiB default.rgw.meta 19 N/A N/A 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182 B 9.0 KiB default.rgw.log 20 N/A N/A 0 B 0 50 GiB 107 107 307 KiB 205 KiB 0 B default.rgw.control 21 N/A N/A 0 B 0 50 GiB 5 5 0 B 0 B 0 B default.rgw.buckets.index 22 N/A N/A 0 B 0 50 GiB 5 5 501 B 60 B 0 B default.rgw.buckets.data 23 N/A N/A 377 MiB 0.39 50 GiB 160 160 385 B 1.1 KiB 603 MiB [root@server1 ~]# [root@server1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server1 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.0 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.1 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.2 -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 0 - host server2 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.3 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.4 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 1.04 99 osd.5 -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server3 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 0.96 78 osd.6 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.7 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.8 TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 MIN/MAX VAR: 0/1.04 STDDEV: 5.58 [root@server1 ~]# I'm kind of newbie to ceph, so any help or hint would be appreciated. Did I hit a bug or something is wrong with my configuration? Thanks a lot, Khodayar

5 5

Ceph Snapshot Children not exists / children relation broken
by Torsten Ennenbach 20 Aug '20

20 Aug '20

Hello all, I`ve a problem with an undeletable Image. Let my try to explain. There is a storage with a Snapshot, and the snapshot thinks he has a Children: rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd' 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16 2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16rbd: unprotecting snap failed: (16) Device or resource busy rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error looking up name for image id f907bc6b8b4567 in pool rbd rbd: listing children failed: (2) No such file or directory So i am not able to unprotect and delete the Snapshot. Is there a way to fix this issue? I know this happens if deep-flatten is not enabled, and this is the root cause I think. We`ve activated this feature now but this s artifact from a time where it was disabled. Best regards & thx for your help. Torsten

3 8

Help
by Randy Morgan 17 Aug '20

17 Aug '20

We are seeking information on configuring Ceph to work with Noobaa and NextCloud. Randy -- Randy Morgan CSR Department of Chemistry/BioChemistry Brigham Young University randym(a)chem.byu.edu

3 2

Ceph Nautilus packages for Ubuntu Focal
by Stefan Kooman 17 Aug '20

17 Aug '20

Hi list, We're wondering if Ceph Nautilus packages will be provided for Ubuntu Focal Fossa (20.04)? You might wonder why one would not just use Ubuntu Bionic (18.04) instead of using the latest LTS. Here is why: a glibc bug in Ubuntu Bionic that *might* affect Open vSwitch (OVS) users [1]. We had quite a few issues with OVS deadlocks on hypervisors, and do not want to risk experiencing the same issues on our Ceph cluster(s). I'm not sure how many of you use OVS for bridging / bonding, but for those who do, running Ceph (Nautlilus / Octopus) on 20.04 would be preferred. Gr. Stefan [1]: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592

2 1

Ceph Tech Talk: A Different Scale – Running small ceph clusters in multiple data centers by Yuval Freund
by Mike Perez 13 Aug '20

13 Aug '20

Hi everyone, Get ready for July 23rd at 17:00 UTC another Ceph Tech Talk but a different scale: running small Ceph clusters in multiple data centers. Thanks to Yuval Freund for giving time to provide us content this month. You can find the calendar invite and archive here: https://ceph.io/ceph-tech-talks/ -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 1

Get World-Class Programming Assignment Help at a Reasonable budget.
by keithmorris081＠gmail.com 12 Aug '20

12 Aug '20

Get it completed from the Top notch Assignment Writers for the Students across Universities across various countries. With the best in class turnaround time and also avail amazing value-added service, And Get in touch with the Best Programming Assignment Help who are very talented in providing solutions on various topics on subjects. You Read it Right! We are here to help you with your queries on your assignments. We are providing you with a world-class and Highly Talented Assignment expert. These Assignment Writers who are capable of doing any writing services, and many other writing services such as Assignment Writing Services, Assignment Help for various subjects such as Engineering Assignment, Statistics for the students across Australia, and also for students across UK, USA, Malaysia and various other countries. https://www.myassignmentservices.com/programming-assignment-help.html

2 1

snaptrim blocks IO on ceph nautilus
by Manuel Lausch 06 Aug '20

06 Aug '20

Hi, since some days I try to debug a problem with snaptrimming under nautilus. I have a cluster with Nautilus (v14.2.10) , 44 Nodes á 24 OSDs á 14 TB I create every day a snapshot for 7 days. Every time the old snapshot is deleting I have bad IO performcance and blocked requests for several seconds until the snaptrim is done. Settings like snaptrim_sleep and osd_pg_max_concurrent_snap_trims don't affect this behavior. In the debug_osd 10/10 log I see the following: 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edda20 prio 196 cost 0 latency 0.019545 osd_repop_reply(client.22731418.0:615257 3.636 e22457/22372) v2 pg pg[3.636( v 22457'100855 (21737'97756,22457'100855] local-lis/les=22372/22374 n=27762 ec=2842/2839 lis/c 22372/22372 les/c/f 22374/22374/0 22372/22372/22343) [411,36,956,763] r=0 lpr=22372 luod=22457'100854 crt=22457'100855 lcod 22457'100853 mlcod 22457'100853 active+clean+snaptrim_wait trimq=[1d~1]] 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edda20 finish 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edc2c0 prio 127 cost 0 latency 0.043165 MOSDScrubReserve(2.2645 RELEASE e22457) v1 pg pg[2.2645( empty local-lis/les=22359/22364 n=0 ec=2403/2403 lis/c 22359/22359 les/c/f 22364/22367/0 22359/22359/22359) [379,411,884,975] r=1 lpr=22359 crt=0'0 active mbc={}] 2020-07-27 11:45:49.976 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557886edc2c0 finish 2020-07-27 11:45:50.039 7fd8b8404700 10 osd.411 pg_epoch: 22457 pg[3.278e( v 22457'99491 (21594'96426,22457'99491] local-lis/les=22359/22362 n=27669 ec=2859/2839 lis/c 22359/22359 les/c/f 22362/22365/0 22359/22359/22343) [411,379,848,924] r=0 lpr=22359 crt=22457'99491 lcod 22457'99489 mlcod 22457'99489 active+clean+snaptrim trimq=[1d~1]] snap_trimmer posting 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 pg_epoch: 22457 pg[3.278e( v 22457'99493 (21594'96426,22457'99493] local-lis/les=22359/22362 n=27669 ec=2859/2839 lis/c 22359/22359 les/c/f 22362/22365/0 22359/22359/22343) [411,379,848,924] r=0 lpr=22359 luod=22457'99491 crt=22457'99493 lcod 22457'99489 mlcod 22457'99489 active+clean+snaptrim trimq=[1d~1]] snap_trimmer complete 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557880ac3760 prio 127 cost 663 latency 7.761823 osd_repop(osd.217.0:3025 3.1ca5 e22457/22378) v2 pg pg[3.1ca5( v 22457'100370 (21716'97357,22457'100370] local-lis/les=22378/22379 n=27532 ec=2855/2839 lis/c 22378/22378 les/c/f 22379/22379/0 22378/22378/22378) [217,411,551,1055] r=1 lpr=22378 luod=0'0 lua=22294'100006 crt=22457'100370 lcod 22457'100369 active mbc={}] 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x557880ac3760 finish 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x5578813e1e40 prio 127 cost 0 latency 7.494296 MOSDScrubReserve(2.37e2 REQUEST e22457) v1 pg pg[2.37e2( empty local-lis/les=22355/22356 n=0 ec=2412/2412 lis/c 22355/22355 les/c/f 22356/22356/0 22355/22355/22355) [245,411,834,768] r=1 lpr=22355 crt=0'0 active mbc={}] 2020-07-27 11:45:57.801 7fd8b8404700 10 osd.411 22457 dequeue_op 0x5578813e1e40 finish the dequeueing of ops works without pauses until the „snap_trimmer posting“ and „snap_trimmer complete“ loglines. This task takes in this example about 7 Seconds. The following operations which are dequeued have now a latency of about this time. I tried to drill down this in the code. (Developers are asked here) It seems, that the PG will be locked for every operation. The snap_trimmer posting and complete message comes from „osd/PrimaryLogPG.cc“ on line 4700. This indicates me, that the process of deleting a snapshot object will sometimes take some time. After further poking around. I see in „osd/SnapMapper.cc“ the method „SnapMapper::get_next_objects_to_trim“ which takes several seconds to get finished. I followed this further to the „common/map_cacher.hpp“ to the line 94: „int r = driver->get_next(key, &store);“ From there I lost the path. The slowness is not on all OSDs at the same time. Somteime, this few OSDs are affected, sometimes some others. Restart of an OSD does not help. With luminous and filestore, snapshot deletion was not an issue at all. With nautilus and bluestore this is not acceptable for my usecase. I don‘t know so far, if this is a bluestore specific problem or some general issue. I wonder a bit why there are no other who have this problem. Regards Manuel

1 2

Ceph influxDB support versus Telegraf Ceph plugin?
by victorhooi＠yahoo.com 06 Aug '20

06 Aug '20

Hi, I've read that Ceph has some InfluxDB reporting capabilities inbuilt (https://docs.ceph.com/docs/master/mgr/influx/) However, Telegraf, which is the system reporting daemon for InfluxDB, also has a Ceph plugin (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/ceph) Just curious what people's thoughts on the two are, or what they are using in production? Which is easier to deploy/maintain, have you found? Or more useful for alerting, or tracking performance gremlins? Thanks, Victor

2 2

RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade
by Chris Palmer 05 Aug '20

05 Aug '20

I have an RGW bucket (backups) that is versioned. A nightly job creates a new version of a few objects. There is a lifecycle policy (see below) that keeps 18 days of versions. This has been working perfectly and has not been changed. Until I upgraded Octopus... The nightly job creates separate log files, including a listing of the object versions. From these I can see that: 13/7 02:14 versions from 13/7 01:13 back to 24/6 01:17 (correct) 14/7 02:14 versions from 14/7 01:13 back to 25/6 01:14 (correct) 14/7 10:00 upgrade Octopus 15.2.3 -> 15.2.4 15/7 02:14 versions from 15/7 01:13 back to 25/6 01:14 (would have expected 25/6 to have expired) 16/7 02:14 versions from 16/7 01:13 back to 15/7 01:13 (now all pre-upgrade versions have wrongly disappeared) It's not a big deal for me as they are only backups, providing it continues to work correctly from now on. However it may affect some other people much more. Any ideas on the root cause? And if it is likely to be stable again now? Thanks, Chris { "Rules": [ { "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "Expiration & incomplete uploads", "Prefix": "", "Status": "Enabled", "NoncurrentVersionExpiration": { "NoncurrentDays": 18 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } } ] }

4 9

Ldap Integration
by Jared Jacob 04 Aug '20

04 Aug '20

Hello, I am looking into connecting my rados gateway to LDAP and found the following documentation. https://docs.ceph.com/docs/master/radosgw/ldap-auth/ I would like to allow an LDAP group to have access to create and manage buckets. The questions I still have are the following: -Do the LDAP users need to log in to some sort of portal before their corresponding ceph user is created? If so, where do they go to do so? Or does the creation of ceph users and keys happen automatically? -How can you access the ldap users key and secret after they are integrated? Thanks in advance for any information you can provide. Regards, Jared

3 4

mimic: much more raw used than reported
by Frank Schilder 03 Aug '20

03 Aug '20

Dear fellow cephers, I observe a wired problem on our mimic-13.2.8 cluster. We have an EC RBD pool backed by HDDs. These disks are not in any other pool. I noticed that the total capacity (=USED+MAX AVAIL) reported by "ceph df detail" has shrunk recently from 300TiB to 200TiB. Part but by no means all of this can be explained by imbalance of the data distribution. When I compare the output of "ceph df detail" and "ceph osd df tree", I find 69TiB raw capacity used but not accounted for; see calculations below. These 69TiB raw are equivalent to 20% usable capacity and I really need it back. Together with the imbalance, we loose about 30% capacity. What is using these extra 69TiB and how can I get it back? Some findings: These are the 5 largest images in the pool, accounting for a total of 97TiB out of 119TiB usage: # rbd du : NAME PROVISIONED USED one-133 25 TiB 14 TiB NAME PROVISIONED USED one-153@222 40 TiB 14 TiB one-153@228 40 TiB 357 GiB one-153@235 40 TiB 797 GiB one-153@241 40 TiB 509 GiB one-153@242 40 TiB 43 GiB one-153@243 40 TiB 16 MiB one-153@244 40 TiB 16 MiB one-153@245 40 TiB 324 MiB one-153@246 40 TiB 276 MiB one-153@247 40 TiB 96 MiB one-153@248 40 TiB 138 GiB one-153@249 40 TiB 1.8 GiB one-153@250 40 TiB 0 B one-153 40 TiB 204 MiB <TOTAL> 40 TiB 16 TiB NAME PROVISIONED USED one-391@3 40 TiB 432 MiB one-391@9 40 TiB 26 GiB one-391@15 40 TiB 90 GiB one-391@16 40 TiB 0 B one-391@17 40 TiB 0 B one-391@18 40 TiB 0 B one-391@19 40 TiB 0 B one-391@20 40 TiB 3.5 TiB one-391@21 40 TiB 5.4 TiB one-391@22 40 TiB 5.8 TiB one-391@23 40 TiB 8.4 TiB one-391@24 40 TiB 1.4 TiB one-391 40 TiB 2.2 TiB <TOTAL> 40 TiB 27 TiB NAME PROVISIONED USED one-394@3 70 TiB 1.4 TiB one-394@9 70 TiB 2.5 TiB one-394@15 70 TiB 20 GiB one-394@16 70 TiB 0 B one-394@17 70 TiB 0 B one-394@18 70 TiB 0 B one-394@19 70 TiB 383 GiB one-394@20 70 TiB 3.3 TiB one-394@21 70 TiB 5.0 TiB one-394@22 70 TiB 5.0 TiB one-394@23 70 TiB 9.0 TiB one-394@24 70 TiB 1.6 TiB one-394 70 TiB 2.5 TiB <TOTAL> 70 TiB 31 TiB NAME PROVISIONED USED one-434 25 TiB 9.1 TiB The large 70TiB images one-391 and one-394 are currently copied to with ca. 5TiB per day. Output of "ceph df detail" with some columns removed: NAME ID USED %USED MAX AVAIL OBJECTS RAW USED sr-rbd-data-one-hdd 11 119 TiB 58.45 84 TiB 31286554 158 TiB Pool is EC 6+2. USED is correct: 31286554*4MiB=119TiB. RAW USED is correct: 119*8/6=158TiB. Most of this data is freshly copied onto large RBD images. Compression is enabled on this pool (aggressive,snappy). However, when looking at "deph osd df tree", I get The combined raw capacity of OSDs backing this pool is 406.8TiB (sum over SIZE). Summing up column USE over all OSDs gives 227.5TiB. This gives a difference of 69TiB (=227-158) that is not accounted for. Here the output of "ceph osd df tree limited" to the drives backing the pool: ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME 84 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 180 MiB 16 GiB 3.9 TiB 56.43 1.72 103 osd.84 145 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.6 TiB 144 MiB 14 GiB 4.3 TiB 51.37 1.57 87 osd.145 156 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.1 TiB 173 MiB 16 GiB 3.8 TiB 57.91 1.77 100 osd.156 168 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 164 MiB 16 GiB 3.9 TiB 56.31 1.72 98 osd.168 181 hdd 8.90999 1.00000 8.9 TiB 5.5 TiB 5.4 TiB 121 MiB 17 GiB 3.5 TiB 61.26 1.87 105 osd.181 74 hdd 8.90999 1.00000 8.9 TiB 4.2 TiB 4.2 TiB 148 MiB 13 GiB 4.7 TiB 46.79 1.43 85 osd.74 144 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 106 MiB 15 GiB 4.2 TiB 53.17 1.62 94 osd.144 157 hdd 8.90999 1.00000 8.9 TiB 5.8 TiB 5.8 TiB 192 MiB 18 GiB 3.1 TiB 65.02 1.99 111 osd.157 169 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 172 MiB 16 GiB 3.8 TiB 56.99 1.74 102 osd.169 180 hdd 8.90999 1.00000 8.9 TiB 5.8 TiB 5.8 TiB 131 MiB 18 GiB 3.1 TiB 65.04 1.99 111 osd.180 60 hdd 8.90999 1.00000 8.9 TiB 4.5 TiB 4.5 TiB 155 MiB 14 GiB 4.4 TiB 50.40 1.54 93 osd.60 146 hdd 8.90999 1.00000 8.9 TiB 4.8 TiB 4.8 TiB 139 MiB 15 GiB 4.1 TiB 53.70 1.64 92 osd.146 158 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.5 TiB 183 MiB 17 GiB 3.4 TiB 62.30 1.90 109 osd.158 170 hdd 8.90999 1.00000 8.9 TiB 5.7 TiB 5.6 TiB 205 MiB 18 GiB 3.2 TiB 63.53 1.94 112 osd.170 182 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.6 TiB 105 MiB 14 GiB 4.3 TiB 52.27 1.60 92 osd.182 63 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 156 MiB 15 GiB 4.2 TiB 52.74 1.61 98 osd.63 148 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.1 TiB 119 MiB 16 GiB 3.8 TiB 57.82 1.77 100 osd.148 159 hdd 8.90999 1.00000 8.9 TiB 4.0 TiB 4.0 TiB 89 MiB 12 GiB 4.9 TiB 44.61 1.36 79 osd.159 172 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 173 MiB 16 GiB 3.8 TiB 57.22 1.75 98 osd.172 183 hdd 8.90999 1.00000 8.9 TiB 6.0 TiB 6.0 TiB 135 MiB 19 GiB 2.9 TiB 67.35 2.06 118 osd.183 229 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.6 TiB 127 MiB 15 GiB 4.3 TiB 52.05 1.59 93 osd.229 232 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.2 TiB 158 MiB 17 GiB 3.7 TiB 58.22 1.78 101 osd.232 235 hdd 8.90999 1.00000 8.9 TiB 4.1 TiB 4.1 TiB 103 MiB 13 GiB 4.8 TiB 45.96 1.40 79 osd.235 238 hdd 8.90999 1.00000 8.9 TiB 5.4 TiB 5.4 TiB 120 MiB 17 GiB 3.5 TiB 60.47 1.85 104 osd.238 259 hdd 10.91399 1.00000 11 TiB 6.2 TiB 6.2 TiB 140 MiB 19 GiB 4.7 TiB 56.54 1.73 120 osd.259 231 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 114 MiB 16 GiB 3.8 TiB 56.90 1.74 101 osd.231 233 hdd 8.90999 1.00000 8.9 TiB 5.5 TiB 5.5 TiB 123 MiB 17 GiB 3.4 TiB 61.78 1.89 106 osd.233 236 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 114 MiB 16 GiB 3.8 TiB 57.53 1.76 101 osd.236 239 hdd 8.90999 1.00000 8.9 TiB 4.2 TiB 4.2 TiB 95 MiB 13 GiB 4.7 TiB 47.41 1.45 86 osd.239 263 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.3 TiB 178 MiB 17 GiB 5.6 TiB 48.73 1.49 102 osd.263 228 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 113 MiB 16 GiB 3.8 TiB 57.10 1.74 96 osd.228 230 hdd 8.90999 1.00000 8.9 TiB 4.9 TiB 4.9 TiB 144 MiB 16 GiB 4.0 TiB 55.20 1.69 99 osd.230 234 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.6 TiB 164 MiB 18 GiB 3.3 TiB 63.29 1.93 109 osd.234 237 hdd 8.90999 1.00000 8.9 TiB 4.8 TiB 4.8 TiB 110 MiB 15 GiB 4.1 TiB 54.33 1.66 97 osd.237 260 hdd 10.91399 1.00000 11 TiB 5.4 TiB 5.4 TiB 152 MiB 17 GiB 5.5 TiB 49.35 1.51 104 osd.260 0 hdd 8.90999 1.00000 8.9 TiB 5.2 TiB 5.2 TiB 157 MiB 16 GiB 3.7 TiB 58.28 1.78 102 osd.0 2 hdd 8.90999 1.00000 8.9 TiB 5.3 TiB 5.2 TiB 122 MiB 16 GiB 3.6 TiB 59.05 1.80 106 osd.2 72 hdd 8.90999 1.00000 8.9 TiB 4.4 TiB 4.4 TiB 145 MiB 14 GiB 4.5 TiB 49.89 1.52 89 osd.72 76 hdd 8.90999 1.00000 8.9 TiB 5.1 TiB 5.1 TiB 178 MiB 16 GiB 3.8 TiB 56.89 1.74 102 osd.76 86 hdd 8.90999 1.00000 8.9 TiB 4.6 TiB 4.5 TiB 155 MiB 14 GiB 4.3 TiB 51.18 1.56 94 osd.86 1 hdd 8.90999 1.00000 8.9 TiB 4.9 TiB 4.9 TiB 141 MiB 15 GiB 4.0 TiB 54.73 1.67 95 osd.1 3 hdd 8.90999 1.00000 8.9 TiB 4.7 TiB 4.7 TiB 156 MiB 15 GiB 4.2 TiB 52.40 1.60 94 osd.3 73 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 4.9 TiB 146 MiB 16 GiB 3.9 TiB 55.68 1.70 102 osd.73 85 hdd 8.90999 1.00000 8.9 TiB 5.6 TiB 5.5 TiB 192 MiB 18 GiB 3.3 TiB 62.46 1.91 109 osd.85 87 hdd 8.90999 1.00000 8.9 TiB 5.0 TiB 5.0 TiB 189 MiB 16 GiB 3.9 TiB 55.91 1.71 102 osd.87 Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 12

Problems with long taking deep-scrubbing processes causing PG_NOT_DEEP_SCRUBBED
by Carsten Grommel - Profihost AG 03 Aug '20

03 Aug '20

Hi, we are having problems with really long taking deep-scrubb processes causing PG_NOT_DEEP_SCRUBBED and ceph HEALTH_WARN. One ph is waiting since 2020-05-18 for the deep-scrubb. Is there any way to speed up the deep-scrubbing? Ceph-Version: ceph version 14.2.8-3-gc6b8eedb77 (c6b8eedb771089fe3b0a95da93158ec4144758f3) nautilus (stable)

3 4

What are smart ways to fix HP printer offline?
by masonlava77＠gmail.com 31 Jul '20

31 Jul '20

HP printer offline is one of the most common errors experienced by countless users. This offline problem is showing a warning message that printing machine does not print any kind of document. I am passing through technical issue from the last week. I don’t have solutions for this offline error, hence my all printing tasks are pending. Why does my HP printer keep going offline? I look for expert’s assistance. So anyone can guide me properly for solving this issue. https://www.hpprintersupportpro.com/blog/stop-my-hp-printer-from-going-offl…

1 0

HP Officejet pro 802 setup
by cfaray3536＠gmail.com 31 Jul '20

31 Jul '20

Get the quick solution to establish the hp Officejet pro 8028 setups without any interruptions. However, if you still have issues or errors and you are stuck in between the process then all you need to do is to visit the HP website and get assistance from the customer support team https://hp-printer-support.us/blog/hp-officejet-pro-8028-setup/

1 0

What is Mcafee Login?
by sophia.smith0321＠gmail.com 31 Jul '20

31 Jul '20

Mcafee is a globally acclaimed brand that gives unbeatable cybersecurity solutions for all of your connected device. It provides the simplest of the breed software applications and tools that reach holistic and robust protection. These innovative products are an ideal blend of avant-garde technology, intuitive features and unsurpassable performance. This user-friendly feature-packed software guarantees unmatchable user experience. It further elevates its overall experience by granting complete control to users through an extremely intuitive <a href="https://mcafeecomactivates.com/mcafee-login/">McAfee Login</a> account. Mcafee Login Account <a href="https://mcafeecomactivates.com/mcafee-login/">McAfee Account Login</a> is your one-stop key for accessing and utilising all the advantages and features of the amazing Mcafee security applications. This online platform allows you to purchase and manage your subscriptions with utmost ease. Besides, it allows you to manage your account and billing information. You can purchase and add new products; register, activate and renew subscriptions; update billing and profile details; check purchase history, add devices and a lot more. In short, it grants you complete control of your service and its features.

1 0

unbalanced pg/osd allocation
by Budai Laszlo 31 Jul '20

31 Jul '20

Dear all, We have a ceph cluster where we are have configured two SSD only pools in order to use them as cache tier for the spinning discs. Altogether there are 27 SSDs organized on 9 hosts distributed in 3 chassis. The hierarchy looks like this: $ ceph osd df tree | grep -E 'ssd|ID' ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME -40 8.26199 - 8.26TiB 5.78TiB 2.48TiB 70.02 5.77 - root ssd-root -50 2.75400 - 2.75TiB 1.93TiB 845GiB 70.02 5.77 - chassis c1-ssd -41 0.91800 - 940GiB 651GiB 289GiB 69.23 5.71 - host c1-h01-ssd 110 ssd 0.30600 1.00000 313GiB 199GiB 115GiB 63.37 5.22 77 osd.110 116 ssd 0.30600 1.00000 313GiB 219GiB 94.3GiB 69.91 5.76 89 osd.116 119 ssd 0.30600 1.00000 313GiB 233GiB 80.2GiB 74.41 6.13 87 osd.119 -42 0.91800 - 940GiB 701GiB 239GiB 74.61 6.15 - host c1-h02-ssd 112 ssd 0.30600 1.00000 313GiB 228GiB 84.9GiB 72.91 6.01 85 osd.112 117 ssd 0.30600 1.00000 313GiB 245GiB 67.9GiB 78.32 6.46 97 osd.117 122 ssd 0.30600 1.00000 313GiB 227GiB 85.8GiB 72.61 5.99 87 osd.122 -43 0.91800 - 940GiB 622GiB 318GiB 66.21 5.46 - host c1-h03-ssd 109 ssd 0.30600 1.00000 313GiB 192GiB 122GiB 61.15 5.04 77 osd.109 115 ssd 0.30600 1.00000 313GiB 206GiB 107GiB 65.79 5.42 79 osd.115 120 ssd 0.30600 1.00000 313GiB 225GiB 88.7GiB 71.70 5.91 90 osd.120 -51 2.75400 - 2.75TiB 1.93TiB 845GiB 70.02 5.77 - chassis c2-ssd -46 0.91800 - 940GiB 651GiB 288GiB 69.31 5.71 - host c2-h01-ssd 125 ssd 0.30600 1.00000 313GiB 211GiB 103GiB 67.22 5.54 81 osd.125 130 ssd 0.30600 1.00000 313GiB 233GiB 80.4GiB 74.33 6.13 89 osd.130 132 ssd 0.30600 1.00000 313GiB 208GiB 105GiB 66.38 5.47 79 osd.132 -45 0.91800 - 940GiB 672GiB 267GiB 71.54 5.90 - host c2-h02-ssd 126 ssd 0.30600 1.00000 313GiB 216GiB 97.4GiB 68.90 5.68 87 osd.126 129 ssd 0.30600 1.00000 313GiB 207GiB 106GiB 66.12 5.45 80 osd.129 134 ssd 0.30600 1.00000 313GiB 249GiB 63.9GiB 79.61 6.56 99 osd.134 -44 0.91800 - 940GiB 650GiB 289GiB 69.20 5.70 - host c2-h03-ssd 123 ssd 0.30600 1.00000 313GiB 201GiB 112GiB 64.23 5.29 76 osd.123 127 ssd 0.30600 1.00000 313GiB 217GiB 96.1GiB 69.31 5.71 85 osd.127 131 ssd 0.30600 1.00000 313GiB 232GiB 81.2GiB 74.06 6.11 92 osd.131 -52 2.75400 - 2.75TiB 1.93TiB 845GiB 70.02 5.77 - chassis c3-ssd -47 0.91800 - 940GiB 628GiB 311GiB 66.86 5.51 - host c3-h01-ssd 124 ssd 0.30600 1.00000 313GiB 204GiB 109GiB 65.13 5.37 78 osd.124 128 ssd 0.30600 1.00000 313GiB 202GiB 111GiB 64.59 5.32 76 osd.128 133 ssd 0.30600 1.00000 313GiB 222GiB 91.3GiB 70.86 5.84 86 osd.133 -48 0.91800 - 940GiB 628GiB 312GiB 66.80 5.51 - host c3-h02-ssd 108 ssd 0.30600 1.00000 313GiB 220GiB 92.9GiB 70.35 5.80 86 osd.108 114 ssd 0.30600 1.00000 313GiB 209GiB 105GiB 66.58 5.49 82 osd.114 121 ssd 0.30600 1.00000 313GiB 199GiB 114GiB 63.46 5.23 79 osd.121 -49 0.91800 - 940GiB 718GiB 222GiB 76.40 6.30 - host c3-h03-ssd 111 ssd 0.30600 1.00000 313GiB 219GiB 94.4GiB 69.87 5.76 84 osd.111 113 ssd 0.30600 1.00000 313GiB 241GiB 72.2GiB 76.95 6.34 96 osd.113 118 ssd 0.30600 1.00000 313GiB 258GiB 55.2GiB 82.39 6.79 101 osd.118 The rule used for the two pools is the following: { "rule_id": 1, "rule_name": "ssd", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -40, "item_name": "ssd-root" }, { "op": "chooseleaf_firstn", "num": 0, "type": "chassis" }, { "op": "emit" } ] } both pools have the size 3, and the total number of PGs is 768 (256+512). As you can see from the previous table (the PG column) there is a significant difference between the OSD with the largest number of PGs (101PGs on osd.118) and the ones with the smallest number (76 PGs on osd.123). The ratio between the two is 1.32. So OSD 118 has more chances to receive data then OSD 123, and we can see that indeed osd.118 is the one storing the most data (82.39% full in the above table). I would like to re balance the PG/OSD allocation. I know that I can play around with the OSD weights (currently .306 for all the OSDs), but I wonder if there is any drawback for this on the long run? Are you aware of any reason why I should NOT modify the weights (and leave those modifications permanent)? Any ideas are welcome :) Kind regards, Laszlo

2 2

Fix tech jumble up in Cash App? Contact help gathering and Talk to a Cash App Representative.
by mary smith 31 Jul '20

31 Jul '20

Any application isn't sans botch and thusly if you face an issue with the application, by then you ought to go to the assistance locales to get your issue isolated and fixed. You can likewise use another other choice and that is to dial the help number and get the issue fixed by choosing to Talk to a Cash App Representative. https://www.customercare-email.com/customer-service/cash-app.html

1 0

Issue in the Epson interface? Discover support from Epson Customer Service.
by mary smith 31 Jul '20

31 Jul '20

If you face an interface issue in your device, by then you ought to reboot it. In any case, if the issue doesn't get disentangled, by then you can take help from the tech consultancies and use their exploring strategies. You can similarly contact the Epson Customer Service for discovering support. https://www.epsonprintersupportpro.net/

1 0

Migrating to managed OSDs with ceph orch
by lstockner＠genesiscloud.com 30 Jul '20

30 Jul '20

Hi all, I just updated a Ceph cluster from Nautilus to Octopus and followed the documentation in order to migrate from the original ceph-ansible setup to cephadm. Overall, this worked great, but there's one part that I couldn't figure out yet and that doesn't seem to be documented: How do I migrate the OSDs to the new managed approach using service specifications? Currently, "ceph orch ps" shows me each OSD and "ceph orch ls" lists them as "osd.2", with "9/0" running with unmanaged placement (iirc osd.2 was the first one I adopted so that's probably where the name comes from). I tried writing a service specification that should match the current deployment and applying that, but the new entries are just sitting there at 0/3 running. For node-exporter, I solved this problem by just removing the old containers and services manually and waiting for Ceph to recreate the new ones, but for OSDs that approach doesn't really seem practical (unless it was a matter of just stopping/removing the old container, but that doesn't seem to do the trick in my tests). Is there a proper way to do this? Or is the cluster just stuck with unmanaged OSDs if it was created without cephadm? Thanks, Lukas

1 0

ceph-ansible epel repo
by Szabo, Istvan (Agoda) 30 Jul '20

30 Jul '20

Hi, I've configured the all.yml like this: grep -v "^#\|^$" ceph/ceph-ansible-playbooks/ceph-ansible-stable-4.0/group_vars/all.yml --- dummy: nautilus: 14 cluster: ceph mon_group_name: mons osd_group_name: osds mgr_group_name: mgrs configure_firewall: False centos_package_dependencies: - epel-release - libselinux-python ceph_origin: repository ceph_repository: community ceph_mirror: http://hk-repo-2001/repo/ceph/ ceph_stable_key: http://hk-repo-2001/repo/ceph/release.asc ceph_stable_release: nautilus ceph_stable_redhat_distro: el7 monitor_interface: bond0 ip_version: ipv4 public_network: 10.121.58.0/24 cluster_network: 192.168.58.0/24 osd_objectstore: bluestore dashboard_enabled: False I'd like to install from our repo server, but when it wants to install at this step it complains about can't reach the epel repository: TASK [ceph-common : install redhat ceph packages] ******************************************************************************************************** FAILED - RETRYING: install redhat ceph packages (3 retries left). FAILED - RETRYING: install redhat ceph packages (3 retries left). FAILED - RETRYING: install redhat ceph packages (3 retries left). FAILED - RETRYING: install redhat ceph packages (2 retries left). FAILED - RETRYING: install redhat ceph packages (2 retries left). FAILED - RETRYING: install redhat ceph packages (2 retries left). FAILED - RETRYING: install redhat ceph packages (1 retries left). FAILED - RETRYING: install redhat ceph packages (1 retries left). FAILED - RETRYING: install redhat ceph packages (1 retries left). fatal: [hk-ceph-2c09]: FAILED! => {"attempts": 3, "changed": false, "msg": "Failure talking to yum: Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again"} fatal: [hk-ceph-2c08]: FAILED! => {"attempts": 3, "changed": false, "msg": "Failure talking to yum: Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again"} fatal: [hk-ceph-2c10]: FAILED! => {"attempts": 3, "changed": false, "msg": "Failure talking to yum: Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again"} FAILED - RETRYING: install redhat ceph packages (3 retries left). FAILED - RETRYING: install redhat ceph packages (3 retries left). FAILED - RETRYING: install redhat ceph packages (2 retries left). FAILED - RETRYING: install redhat ceph packages (3 retries left). FAILED - RETRYING: install redhat ceph packages (3 retries left). FAILED - RETRYING: install redhat ceph packages (1 retries left). FAILED - RETRYING: install redhat ceph packages (2 retries left). fatal: [hk-ceph-2c07]: FAILED! => {"attempts": 3, "changed": false, "msg": "Failure talking to yum: failure: repodata/repomd.xml from epel: [Errno 256] No more mirrors to try.\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.x…: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed7: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed6: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed6: Network is unreachable\"\nhttp://download.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - \"Failed to connect to 2620:52:3:1:dead:beef:cafe:fed6: Network is unreachable\""} FAILED - RETRYING: install redhat ceph packages (2 retries left). FAILED - RETRYING: install redhat ceph packages (2 retries left). FAILED - RETRYING: install redhat ceph packages (1 retries left). FAILED - RETRYING: install redhat ceph packages (1 retries left). FAILED - RETRYING: install redhat ceph packages (1 retries left). I've tried to change on the hk-ceph-2c07 in the epel.repo file uncomment baseurl and comment out metalink, cleaned yum ... but nothing helped, I guess something is wrong with my internal link. When I try to install something from epel repo on the server, I can, but via ansible I can't. I'd like to deploy ceph with ansible user, the rights are correct. Here is the hosts file for the playbook: [all:vars] ansible_ssh_user=ansible ansible_become=true ansible_become_method=sudo ansible_become_user=root [mons] hk-cephm-2007 hk-cephm-2008 hk-cephm-2009 [mgrs] hk-cephm-2007 hk-cephm-2008 hk-cephm-2009 [osds] hk-ceph-2c07 hk-ceph-2c08 hk-ceph-2c09 hk-ceph-2c10 Why is the repo error or network unreachable? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 1

ceph mgr memory leak
by Frank Ritchie 30 Jul '20

30 Jul '20

Hi all, When running containerized Ceph (Nautilus) is anyone else seeing a constant memory leak in the ceph-mgr pod with constant ms_handle_reset errors in the logs for the backup mgr instance? --- 0 client.0 ms_handle_reset on v2:172.29.1.13:6848/1 0 client.0 ms_handle_reset on v2:172.29.1.13:6848/1 0 client.0 ms_handle_reset on v2:172.29.1.13:6848/1 --- I see a couple of related reports with no activity: https://tracker.ceph.com/issues/36471 https://tracker.ceph.com/issues/40260 and one related merge that doesn't seem to have corrected the issue: https://github.com/ceph/ceph/pull/24233 thx Frank

2 2

[nautilus][mds] MDS fall into ReadOnly mode
by Frank Yu 30 Jul '20

30 Jul '20

Hi guys, I have a ceph cluster with three MDS servers, two of them in active status, while the left one is in standby-replay mode. Today I found the message '1 MDSs are read only' show up when check the cluster status with 'ceph -s', details as below: # ceph -s cluster: id: 3d43e9a5-50dc-4f84-9493-656bf4f06f8c health: HEALTH_WARN 5 clients failing to advance oldest client/flush tid 1 MDSs are read only 2 MDSs report slow requests 2 MDSs behind on trimming BlueFS spillover detected on 33 OSD(s) services: mon: 3 daemons, quorum bjcpu-001,bjcpu-002,bjcpu-003 (age 3M) mgr: bjcpu-001.xxxx.io(active, since 3M), standbys: bjcpu-003.xxxx.io, bjcpu-002.xxxx.io mds: cephfs:2 {0=bjcpu-003.xxxx.io=up:active,1=bjcpu-001.xxxx.io=up:active} 1 up:standby-replay osd: 48 osds: 48 up (since 7w), 48 in (since 7M) data: pools: 3 pools, 2304 pgs objects: 301.35M objects, 70 TiB usage: 246 TiB used, 280 TiB / 527 TiB avail pgs: 2295 active+clean 9 active+clean+scrubbing+deep io: client: 254 B/s rd, 44 MiB/s wr, 0 op/s rd, 15 op/s wr What should I do to fix the error message? it seems the cluster still works fine(can read and write). Many thanks -- Regards Frank Yu

2 4

Not able to access radosgw S3 bucket creation with AWS java SDK. Caused by: java.net.UnknownHostException: issue.
by sathvik vutukuri 30 Jul '20

30 Jul '20

Hi All, radosgw-admin is configured in ceph-deploy, created a few buckets from the Ceph dashboard, but when accessing through Java AWS S3 code to create a new bucket i am facing the below issue.. Exception in thread "main" com.amazonaws.SdkClientException: Unable to execute HTTP request: firstbucket.rgwhost at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008) at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5950) at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1812) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1772) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1710) at org.S3.App.main(App.java:71) Caused by: java.net.UnknownHostException: firstbucket.rgwhost at java.net.InetAddress.getAllByName0(InetAddress.java:1281) at java.net.InetAddress.getAllByName(InetAddress.java:1193) at java.net.InetAddress.getAllByName(InetAddress.java:1127) at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy3.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ... 15 more -- Thanks, Vutukuri Sathvik, 8197748291.

3 5

Reply with Attachments in Outlook
by emailfix11＠gmail.com 30 Jul '20

30 Jul '20

People who frequently deal with emails in Outlook must be aware of a common issue. That is we won’t be able to reply with original attachments. This situation does result in multitudinous troubles. For instance, if in reply we’ve put forward some errors about original attachments, recipients who need to check the errors wouldn’t be able to find and open the attachments directly in our reply. They have to go to “Sent Items” folder and open the original email as well as its inside attachments. Far and away, it is pretty inconvenient for both senders and recipients. In fact, this case isn’t involved with any outlook reply with attachment. It only manifests a default configuration of Outlook. In a nutshell, reply can’t attach original messages by default in Outlook. Nonetheless, we can configure Outlook to permit reply with original attachments. Here are the concrete steps. For more info: https://www.emailsfix.com/outlook-email/how-to-reply-with-attachment-in-out…

1 0

High io wait when osd rocksdb is compacting
by Raffael Bachmann 30 Jul '20

30 Jul '20

Hi All, I'm kind of crossposting this from here: https://forum.proxmox.com/threads/i-o-wait-after-upgrade-5-x-to-6-2-and-cep… But since I'm more and more sure that it's a ceph problem I'll try my luck here. Since updating from Luminous to Nautilus I have a big problem. I have a 3 node cluster. Each cluster has 2 nvme ssd and a 10GBASE-T net for ceph. Every few minutes a osd seems to compact the rocksdb. While doing this it uses alot of I/O and blocks. This basically blocks the whole cluster and no VM/Container can read data for some seconds (minutes). While it happens "iostat -x" looks like this: Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util nvme0n1 0.00 2.00 0.00 24.00 0.00 46.00 0.00 95.83 0.00 0.00 0.00 0.00 12.00 2.00 0.40 nvme1n1 0.00 1495.00 0.00 3924.00 0.00 6099.00 0.00 80.31 0.00 352.39 523.78 0.00 2.62 0.67 100.00 And iotop: Total DISK READ: 0.00 B/s | Total DISK WRITE: 1573.47 K/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 3.43 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 2306 be/4 ceph 0.00 B/s 1533.22 K/s 0.00 % 99.99 % ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph [rocksdb:low1] In the ceph-osd log I see that rocksdb is compacting. https://gist.github.com/qwasli/3bd0c7d535ee462feff8aaee618f3e08 The pool and one OSD is nearfull. I'd planed to move some data away to another ceph pool. But now I'm not sure anymore if I should go with ceph. I'l move some data away anyway today to see if that helps, but before the upgrade there was the same amount of data an I haven't had a problem. Any hints to solve this are appreciated. Cheers Raffael

4 9

cephadm and disk partitions
by Jason Borden 29 Jul '20

29 Jul '20

Greetings, I have a question regarding the use of cephadm and disk partitions. I have noticed that the cephadm documentation mentions that a device cannot have partitions to be considered "available" for use. In my situation I don't want to use a device with partitions, but rather a partition itself as an osd. I've noticed that partitions do not show up when using `ceph orch device ls`. I've also noticed that partitions can still be used as osds by running something like `ceph orch daemon add osd node1:/dev/sda4`. My question is should I? Am I going to run into trouble by using a partition for an osd instead of a full device? Thanks, Jason

3 4

Setting rbd_default_data_pool through the config store
by Wido den Hollander 29 Jul '20

29 Jul '20

Hi, I'm trying to have clients read the 'rbd_default_data_pool' config option from the config store when creating a RBD image. This doesn't seem to work and I'm wondering if somebody knows why. I tried: $ ceph config set client rbd_default_data_pool rbd-data $ ceph config set global rbd_default_data_pool rbd-data They both show up under: $ ceph config dump However, newly created RBD images with the 'rbd' CLI tool do not use the data pool. If I set this in ceph.conf it works: [client] rbd_default_data_pool = rbd-data Somehow librbd isn't fetching these configuration options. Any hints on how to get this working? The end result is that libvirt (which doesn't read ceph.conf) should also be able to create RBD images with a different data pool. Wido

2 6

Usable space vs. Overhead
by David Orman 29 Jul '20

29 Jul '20

I'm having a hard time understanding the EC usable space vs. raw. https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/ indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space, but that's not lining up with what i'd expect just from k data chunks + m parity chunks. So, for example, k=4, m=2. you'd expect every 4 byte object written would consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw storage. What am I missing here?

4 7

Re: S3 bucket lifecycle not deleting old objects
by Robin H. Johnson 29 Jul '20

29 Jul '20

On Tue, Jul 28, 2020 at 01:28:14PM +0000, Alex Hussein-Kershaw wrote: > Hello, > > I have a problem that old versions of S3 objects are not being deleted. Can anyone advise as to why? I'm using Ceph 14.2.9. How many objects are in the bucket? If it's a lot, then you may run into RGW's lifecycle performance limitations: listing each bucket is a very slow operation for lifecycle prior to improvements make in later versions (Octopus with maybe a backport to Nautilius?) If the bucket doesn't have a lot of operations, you could try running the 'radosgw-admin lc process' directly, with debug logging, and see where it gets bogged down. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2(a)gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

2 1

How to change toner in sharp printer
by hirkumispe＠enayu.com 29 Jul '20

29 Jul '20

Don't know how to change toner cartridge on your sharp printer? Check our this step by step guide to learn how to replace toner in sharp printer https://printersetup.org/change-toner-in-sharp-printer/

1 0

How to contact geek squad online
by hirkumispe＠enayu.com 29 Jul '20

29 Jul '20

Fix technical breakdown of all your electronics and appliances at Geek Squad Support. Reach the certified experts at Geek Squad Support for fixing any kind of technical bug with your devices. Best of services and assistance assured at support. https://geekstechs.org/geek-squad-support/ Remove the technical glitches and flaws of your electronic appliances with Geek Squad Online Support. Avail the best online support services for fixing your electronic appliance. Reach the experts at Geek Squad Online Support with 24/7 availability to avail best solutions for your troublesome devices. https://geek-customer-care.com/geek-squad-online-support.html

1 0

Stuck removing osd with orch
by Ml Ml 29 Jul '20

29 Jul '20

Hello, yesterday i did: ceph osd purge 32 --yes-i-really-mean-it I also started to upgrade: ceph orch upgrade start --ceph-version 15.2.4 It seems its really gone: ceph osd crush remove osd.32 => device 'osd.32' does not appear in the crush map ceph orch ps: osd.32 ceph01 error 4h ago 2h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> root@ceph01:~# ceph orch osd rm status NAME HOST PGS STARTED_AT osd.32 ceph01 n/a 2020-07-29 09:50:11.026643 (some hours ago) root@ceph01:~# ceph health detail HEALTH_ERR Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2; Low space hindering backfill (add storage if this doesn't resolve itself): 105 pgs backfill_toofull; Degraded data redundancy: 9882/33167949 objects degraded (0.030%), 1 pg degraded, 1 pg undersized [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2 Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2 Any idea what i would try to remove that OSD? Thanks, Michael

1 0

mon tried to load "000000.sst" which doesn't exist when recovering from osds
by Yu Wei 29 Jul '20

29 Jul '20

Hi, I deployed rook v0.8.3 with ceph 12.2.7. This is production system deployed for a long time. Because unknown reason, mon couldn't form quorum anymore and I tried to restore mon from osd by following document below, https://github.com/ceph/ceph/blob/v12.2.7/doc/rados/troubleshooting/trouble… After collecting cluster map data, replace store.db and restart mon, monitor log indicated that it tried to load "000000.sst" which was not existed. Log also indicated that mon found all .sst files during startup. Detailed log as below. 2020-07-28 09:44:38.100932 I | rook-ceph-mon2: 2020-07-28 09:44:38.100799 7f2e4abd0ec0 4 rocksdb: CURRENT file: CURRENT 2020-07-28 09:44:38.100946 I | rook-ceph-mon2: 2020-07-28 09:44:38.100951 I | rook-ceph-mon2: 2020-07-28 09:44:38.100847 7f2e4abd0ec0 4 rocksdb: IDENTITY file: IDENTITY 2020-07-28 09:44:38.100958 I | rook-ceph-mon2: 2020-07-28 09:44:38.100963 I | rook-ceph-mon2: 2020-07-28 09:44:38.100865 7f2e4abd0ec0 4 rocksdb: MANIFEST file: MANIFEST-000014 size: 284 Bytes 2020-07-28 09:44:38.100967 I | rook-ceph-mon2: 2020-07-28 09:44:38.100972 I | rook-ceph-mon2: 2020-07-28 09:44:38.100869 7f2e4abd0ec0 4 rocksdb: SST files in /var/lib/rook/rook-ceph-mon2/data/store.db dir, Total Num: 3, files: 000004.sst 000007.sst 000010.sst 2020-07-28 09:44:38.100976 I | rook-ceph-mon2: 2020-07-28 09:44:38.100981 I | rook-ceph-mon2: 2020-07-28 09:44:38.100872 7f2e4abd0ec0 4 rocksdb: Write Ahead Log file in /var/lib/rook/rook-ceph-mon2/data/store.db: 000015.log size: 0 ; 2020-07-28 09:44:38.100985 I | rook-ceph-mon2: 2020-07-28 09:44:38.100989 I | rook-ceph-mon2: 2020-07-28 09:44:38.100874 7f2e4abd0ec0 4 rocksdb: Options.error_if_exists: 0 ... 2020-07-28 09:44:38.101528 I | rook-ceph-mon2: 2020-07-28 09:44:38.101467 7f2e4abd0ec0 4 rocksdb: Fast CRC32 supported: 1 2020-07-28 09:44:38.104667 I | rook-ceph-mon2: 2020-07-28 09:44:38.104317 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000014 2020-07-28 09:44:38.104726 I | rook-ceph-mon2: 2020-07-28 09:44:38.104926 I | rook-ceph-mon2: 2020-07-28 09:44:38.104582 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: ... 2020-07-28 09:44:38.105633 I | rook-ceph-mon2: 2020-07-28 09:44:38.104857 7f2e4abd0ec0 4 rocksdb: Options.report_bg_io_stats: 0 2020-07-28 09:44:38.111205 I | rook-ceph-mon2: 2020-07-28 09:44:38.110905 7f2e4abd0ec0 2 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:1062] Unable to load table properties for file 0 --- IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.111266 I | rook-ceph-mon2: 2020-07-28 09:44:38.111693 I | rook-ceph-mon2: 2020-07-28 09:44:38.110999 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/rook/rook-ceph-mon2/data/store.db/MANIFEST-000014 succeeded,manifest_file_number is 14, next_file_number is 17, last_sequence is 111, log_number is 0,prev_log_number is 0,max_column_family is 0 2020-07-28 09:44:38.111723 I | rook-ceph-mon2: 2020-07-28 09:44:38.111732 I | rook-ceph-mon2: 2020-07-28 09:44:38.111006 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 13 2020-07-28 09:44:38.111738 I | rook-ceph-mon2: 2020-07-28 09:44:38.111746 I | rook-ceph-mon2: 2020-07-28 09:44:38.111101 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work 2020-07-28 09:44:38.111764 I | rook-ceph-mon2: 2020-07-28 09:44:38.111214 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:343] Shutdown complete 2020-07-28 09:44:38.112077 I | rook-ceph-mon2: 2020-07-28 09:44:38.111862 7f2e4abd0ec0 -1 rocksdb: Corruption: Can't access /000000.sst: IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.112096 I | rook-ceph-mon2: 2020-07-28 09:44:38.112103 I | rook-ceph-mon2: 2020-07-28 09:44:38.111941 7f2e4abd0ec0 -1 error opening mon data directory at '/var/lib/rook/rook-ceph-mon2/data': (22) Invalid argument 2020-07-28 09:44:38.116813 I | rook-ceph-mon2: 2020-07-28 09:44:38.111862 7f2e4abd0ec0 -1 rocksdb: Corruption: Can't access /000000.sst: IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.116874 I | rook-ceph-mon2: 2020-07-28 09:44:38.116883 I | rook-ceph-mon2: 2020-07-28 09:44:38.111941 7f2e4abd0ec0 -1 error opening mon data directory at '/var/lib/rook/rook-ceph-mon2/data': (22) Invalid argument failed to run mon. failed to start mon: Failed to complete 'rook-ceph-mon2': exit status 1. It's a little weird that it tried to load "000000.sst" although rook found the correct three .sst files during startup. There is no "000000.sst" related content in store.db. Any advice for this problem? Is it possible that I executed any wrong steps? Or is there any workaround for this? root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# ls -al store.db/ total 92 drwxr-xr-x 2 root root 188 Jul 28 01:35 . drwxr--r-- 3 root root 55 Jul 28 02:37 .. -rw-r--r-- 1 root root 56547 Jul 28 02:38 000004.sst -rw-r--r-- 1 root root 1179 Jul 28 02:38 000007.sst -rw-r--r-- 1 root root 1243 Jul 28 02:38 000010.sst -rw-r--r-- 1 root root 0 Jul 28 02:38 000015.log -rw-r--r-- 1 root root 16 Jul 28 02:38 CURRENT -rw-r--r-- 1 root root 37 Jul 28 02:38 IDENTITY -rw-r--r-- 1 root root 0 Jul 28 02:38 LOCK -rw-r--r-- 1 root root 284 Jul 28 02:38 MANIFEST-000014 -rw-r--r-- 1 root root 4620 Jul 28 02:38 OPTIONS-000014 -rw-r--r-- 1 root root 4620 Jul 28 02:38 OPTIONS-000017 root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# find . -type f | xargs grep "000000.sst" root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# find . -type f | xargs grep "000000" ./store.db/OPTIONS-000017: delete_obsolete_files_period_micros=21600000000 ./store.db/OPTIONS-000017: memtable_prefix_bloom_size_ratio=0.000000 ./store.db/OPTIONS-000017: max_bytes_for_level_multiplier=10.000000 ./store.db/OPTIONS-000014: delete_obsolete_files_period_micros=21600000000 ./store.db/OPTIONS-000014: memtable_prefix_bloom_size_ratio=0.000000 ./store.db/OPTIONS-000014: max_bytes_for_level_multiplier=10.000000 Thanks, Jared, (韦煜） Software developer Interested in open source software, big data, Linux

1 0

Does Cash app customer service advise me if an account gets checked?
by david william 29 Jul '20

29 Jul '20

At whatever point you are applying to confirm your character on the Cash app, at that point you will likewise get a warning of further advances. All things considered, you have to check your record, on the off chance that you need to go through more cash, or you need to utilize extra services on it. You just need to enter your complete name, date of birth, last four digits of SSN. In the event that there is an issue, contact Cash app customer service.https://www.cash-app-customer-service.com/

1 0

Issue in getting a Cash App refund on account of headway tab? Call help gathering.
by david william 29 Jul '20

29 Jul '20

The advancement tab is one of the basic holds onto the item this is in all probability subject for the Cash App refund. As such, at the off peril which you have to get your coins back, with the guide of using then you may use the help this is given withinside the assistance targets or you may dial the wide variety to interface with the assist group with anticipating any help. https://www.contact-customer-service.net/blog/cash-app-refund/

1 0

Current best practice for migrating from one EC profile to another?
by David Orman 29 Jul '20

29 Jul '20

Hi, As we expand our cluster (adding nodes), we'd like to take advantage of better EC profiles enabled by higher server/rack counts. I understand, as Ceph currently exists (15.2.4), there is no way to live-migrate from one EC profile to another on an existing pool, for example, from 4+2 to 17+3 when going from 7 nodes to 21. Is this correct? How are people accomplishing migrations such as this one (or 4+2 to 9+3, for example) with minimal disruption to services that utilize RBDs sitting on top of these pools? I found: https://ceph.io/geen-categorie/ceph-pool-migration/ , which requires effectively shutting down access during migration (which is doable, but not ideal) - and from what I've read - has some potential downsides (specifically referencing the cppool method). The pools are all EC for data, and an rbd pool exists for metadata. Thank you!

3 2

July Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 28 Jul '20

28 Jul '20

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday July 22nd. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200722 We try to keep it to an hour or less. Ceph calendar event details: July 22, 2020 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

2 1

repeatable crash in librbd1
by Johannes Naab 28 Jul '20

28 Jul '20

Hi, we observe crashes in librbd1 on specific workloads in virtual machines on Ubuntu 20.04 hosts with librbd1=15.2.4-1focal. The changes in https://github.com/ceph/ceph/commit/50694f790245ca90a3b8a644da7b128a7a148cc6 could be related, but do not easily apply against v15.2.4. We have collected several backtraces, using a reliable local reproducer. (This is a resent of the message stuck in the moderation queue since yesterday.) Best regards Johannes VM in libvirt with: <pre> <disk type='network' device='disk'> <driver name='qemu' type='raw' discard='unmap'/> <source protocol='rbd' name='pool/disk' index='4'>  </source> <iotune> <read_bytes_sec>209715200</read_bytes_sec> <write_bytes_sec>209715200</write_bytes_sec> <read_iops_sec>5000</read_iops_sec> <write_iops_sec>5000</write_iops_sec> <read_bytes_sec_max>314572800</read_bytes_sec_max> <write_bytes_sec_max>314572800</write_bytes_sec_max> <read_iops_sec_max>7500</read_iops_sec_max> <write_iops_sec_max>7500</write_iops_sec_max> <read_bytes_sec_max_length>60</read_bytes_sec_max_length> <write_bytes_sec_max_length>60</write_bytes_sec_max_length> <read_iops_sec_max_length>60</read_iops_sec_max_length> <write_iops_sec_max_length>60</write_iops_sec_max_length> </iotune> </disk> </pre> workload: <pre> fio --rw=write --name=test --size=10M timeout 30s fio --rw=write --name=test --size=20G timeout 3m fio --rw=write --name=test --size=20G --direct=1 timeout 1m fio --rw=randrw --name=test --size=20G --direct=1 timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1 # the backtraces are then observed while the following command is running fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test --size=1G --direct=1 </pre> observed stack traces three times: <pre> #0 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x557f633e9400) at ./src/common/event_socket.h:32 #1 0x00007ffb9740ba34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:262 #2 0x00007ffb9740ce98 in librbd::io::AioCompletion::complete (this=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:104 #3 0x00007ffb9740d1a0 in librbd::io::AioCompletion::complete_request (this=0x557f633f7600, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #4 0x00007ffb9742fdca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x7ffb68364a60, r=4096) at ./src/librbd/io/ReadResult.cc:155 #5 0x00007ffb9728334d in Context::complete (this=0x7ffb68364a60, r=<optimized out>) at ./src/include/Context.h:77 #6 0x00007ffb9742c7d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x7ffb6832bed0, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #7 0x00007ffb9742c735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #8 0x00007ffb9755e942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x7ffb683394a0, r=r@entry=0) at ./src/include/Context.h:78 #9 0x00007ffb97562e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x7ffb683394a0, r=0) at ./src/log/SubsystemMap.h:72 #10 0x00007ffb970f4177 in librados::C_AioComplete::finish (this=0x7ffb68362e30, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #11 0x00007ffb970afd1d in Context::complete (this=0x7ffb68362e30, r=<optimized out>) at ./src/include/Context.h:77 #12 0x00007ffb8765791d in Finisher::finisher_thread_entry (this=0x557f63387a30) at ./src/common/Finisher.cc:66 #13 0x00007ffb9946f609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #14 0x00007ffb99396103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> once: <pre> #0 0x00007fe7a972169d in std::atomic<boost::lockfree::detail::tagged_ptr<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::load (__m=std::memory_order_acquire, this=<optimized out>) at /usr/include/c++/9/atomic:250 #1 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:311 #2 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #3 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x5613db630440) at ./src/librbd/io/AioCompletion.cc:276 #4 0x00007fe7a9721a34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:262 #5 0x00007fe7a9722e98 in librbd::io::AioCompletion::complete (this=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:104 #6 0x00007fe7a97231a0 in librbd::io::AioCompletion::complete_request (this=0x5613dbbfa2c0, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #7 0x00007fe7a9745dca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x5613dc6a5c90, r=4096) at ./src/librbd/io/ReadResult.cc:155 #8 0x00007fe7a959934d in Context::complete (this=0x5613dc6a5c90, r=<optimized out>) at ./src/include/Context.h:77 #9 0x00007fe7a97427d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x5613db8bae40, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #10 0x00007fe7a9742735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #11 0x00007fe7a9874942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x5613dbbfc900, r=r@entry=0) at ./src/include/Context.h:78 #12 0x00007fe7a9878e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x5613dbbfc900, r=0) at ./src/log/SubsystemMap.h:72 #13 0x00007fe7a940a177 in librados::C_AioComplete::finish (this=0x5613db85e6d0, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #14 0x00007fe7a93c5d1d in Context::complete (this=0x5613db85e6d0, r=<optimized out>) at ./src/include/Context.h:77 #15 0x00007fe79b65791d in Finisher::finisher_thread_entry (this=0x5613db369ee0) at ./src/common/Finisher.cc:66 #16 0x00007fe7ab788609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #17 0x00007fe7ab6af103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> twice: <pre> #0 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::extract_ptr (i=<error reading variable>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:113 #1 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::get_ptr (this=0xfffff40ee9c0) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:115 #2 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate_impl<false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:187 #3 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate<true, false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:168 #4 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::construct<true, false, librbd::io::AioCompletion*, boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node*> (arg2=<optimized out>, arg1=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:100 #5 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:302 #6 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #7 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x560f5e10c000) at ./src/librbd/io/AioCompletion.cc:276 #8 0x00007f8c8bcada34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x560f5edd5d30) at ./src/librbd/io/AioCompletion.cc:262 #9 0x00007f8c8bcaee98 in librbd::io::AioCompletion::complete (this=0x560f5edd5d30) at ./src/librbd/io/AioCompletion.cc:104 #10 0x00007f8c8bcaf1a0 in librbd::io::AioCompletion::complete_request (this=0x560f5edd5d30, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #11 0x00007f8c8bcd1dca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x7f8bc4342280, r=4096) at ./src/librbd/io/ReadResult.cc:155 #12 0x00007f8c8bb2534d in Context::complete (this=0x7f8bc4342280, r=<optimized out>) at ./src/include/Context.h:77 #13 0x00007f8c8bcce7d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x7f8bc4326820, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #14 0x00007f8c8bcce735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #15 0x00007f8c8be00942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x7f8bc42b2a30, r=r@entry=0) at ./src/include/Context.h:78 #16 0x00007f8c8be04e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x7f8bc42b2a30, r=0) at ./src/log/SubsystemMap.h:72 #17 0x00007f8c8b996177 in librados::C_AioComplete::finish (this=0x7f8bc43423b0, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #18 0x00007f8c8b951d1d in Context::complete (this=0x7f8bc43423b0, r=<optimized out>) at ./src/include/Context.h:77 #19 0x00007f8c82f4291d in Finisher::finisher_thread_entry (this=0x560f5f44b180) at ./src/common/Finisher.cc:66 #20 0x00007f8c91e1a609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #21 0x00007f8c91d41103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> once: <pre> #0 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::extract_ptr (i=<error reading variable>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:113 #1 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::get_ptr (this=0xfffff40ee9c0) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:115 #2 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate_impl<false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:187 #3 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate<true, false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:168 #4 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::construct<true, false, librbd::io::AioCompletion*, boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node*> (arg2=<optimized out>, arg1=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:100 #5 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:302 #6 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #7 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x55d1b89113c0) at ./src/librbd/io/AioCompletion.cc:276 #8 0x00007f275f6d7a34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x55d1b8a7a0d0) at ./src/librbd/io/AioCompletion.cc:262 #9 0x00007f275f6d8e98 in librbd::io::AioCompletion::complete (this=0x55d1b8a7a0d0) at ./src/librbd/io/AioCompletion.cc:104 #10 0x00007f275f6d91a0 in librbd::io::AioCompletion::complete_request (this=0x55d1b8a7a0d0, r=<optimized out>) at ./src/librbd/io/AioCompletion.cc:229 #11 0x00007f275f54f34d in Context::complete (this=0x7f268833f850, r=<optimized out>) at ./src/include/Context.h:77 #12 0x00007f274f6b647b in ThreadPool::worker (this=0x55d1b9d04860, wt=<optimized out>) at ./src/common/WorkQueue.cc:118 #13 0x00007f274f6b7545 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:466 #14 0x00007f276173e609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #15 0x00007f2761665103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre>

2 6

slow ops on one osd makes all my buckets unavailable
by Zhenshi Zhou 28 Jul '20

28 Jul '20

Hi, My harbor registry uses ceph object storage to save the images. But I couldn't pull/push images from harbor a few moments ago. Ceph was in warning health status in the same time. The cluster just had a warning message said that osd.24 has slow ops. I check the ceph-osd.24.log, and showed as below: *2020-07-28 19:01:40.599 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:41.558 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:42.579 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:43.566 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:44.588 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:45.627 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:46.674 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:47.701 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:48.729 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:49.729 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:50.889 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)......* *......* *2020-07-28 21:03:35.053 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.166298.0:34904067 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)* After restarted osd.24, the cluster became health again, so was harbor. What confuse me is that why my harbor couldn't get data from its bucket while the log indicated that there was a client block on the other bucket. I don't think some slow ops on an osd has any bad effect on all buckets. Any idea is appreciated. Thanks

1 0

S3 bucket lifecycle not deleting old objects
by Alex Hussein-Kershaw 28 Jul '20

28 Jul '20

Hello, I have a problem that old versions of S3 objects are not being deleted. Can anyone advise as to why? I'm using Ceph 14.2.9. I expect old versions of S3 objects to be deleted after 3 days as per my lifecycle config on the bucket: { "Rules": [ { "Status": "Enabled", "Prefix": "", "NoncurrentVersionExpiration": { "NoncurrentDays": 3 }, "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "S3 scsdata bucket: Tidy up old versions" } ] } I have an object with 3 versions below (and is much older than 3 days): [root@hera hera_sdc] /usr/bin> aws s3api --endpoint http://127.3.3.3:7480 list-object-versions --bucket hera-scsdata --key 84/46/2020060508501821902143658709-Subscriber | grep -B 4 -A 6 84/46/2020060508501821902143658709-Subscriber "LastModified": "2020-06-05T08:58:19.644Z", "VersionId": "FUdZIehBu3sgRbNJSmZwj3VHWs1ednH", "ETag": "\"a18286c50a7323efe58497eb97d6dc9d\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": true, "Size": 4440 -- "LastModified": "2020-06-05T08:58:17.943Z", "VersionId": "JVKGMJQS-l7xKQuqdfn4QsEY5WLEosj", "ETag": "\"87e9953af436b702afb80d457f1d73cb\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": false, "Size": 4408 -- "LastModified": "2020-06-05T08:50:19.167Z", "VersionId": "-RSNSCDvGj83f4DZ11s8YZ2KaxT8T.a", "ETag": "\"a68ec68ce825e009ee9a70cfdae9c794\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": false, "Size": 4256 -- ], "NextKeyMarker": "85/49/20200604163626B4C712312312302641-Subscriber", "MaxKeys": 1000, "Prefix": "", "KeyMarker": "84/46/2020060508501821902143658709-Subscriber", "DeleteMarkers": [ { "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, So those objects still being present seems to be in conflict with the config I have set? Thanks, Alex

1 0

Re: Push config to all hosts
by Cem Zafer 28 Jul '20

28 Jul '20

Thanks Ricardo for clarification. Regards. On Mon, Jul 27, 2020 at 2:50 PM Ricardo Marques <RiMarques(a)suse.com> wrote: > Hi Cem, > > Since https://github.com/ceph/ceph/pull/35576 you will be able to tell > cephadm to keep your `/etc/ceph/ceph.conf` updated in all hosts by runnig: > > # ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true > > But this feature was not released yet, so you will have to wait for > v15.2.5. > > > Ricardo Marques > > ------------------------------ > *From:* Cem Zafer <cemzafer(a)gmail.com> > *Sent:* Monday, June 29, 2020 6:37 AM > *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Push config to all hosts > > Hi, > What is the best method(s) to push ceph.conf to all hosts in octopus > (15.x)? > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Weird buckets in a new cluster causing broken dashboard functionality
by Eugen König 28 Jul '20

28 Jul '20

Hi all, I'm currently experience some strange behavior in our cluster: the dashboards object gateway "buckets" submenu is broken and I'm getting 503 errors (however, "Users" and "Daemons" work flawlessly). Looking into the mgr log gives me following error: 2020-07-24T12:38:12.695+0200 7f42150f3700 0 [dashboard ERROR rest_client] RGW REST API failed GET req status: 404 2020-07-24T12:38:12.843+0200 7f42130ef700 0 [dashboard ERROR request] [10.1.0.133:38454] [GET] [500] [0.039s] [eugen] [513.0B] /api/rgw/bucket/91e22800581543b5be4654f7b9b0c7cc_102020-07-24T12:38:12.843+0200 7f42130ef700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "00a3c78a-d96f-4f8b-b3c4-f24eac99f4a1"} '] So it took me to the point to look into buckets and I got this list with weird bucket names (even with dash, but it's actually not allowed): root@ceph1-40-10~# radosgw-admin buckets list [ "12345", "deployment", "91e22800581543b5be4654f7b9b0c7cc_5", "91e22800581543b5be4654f7b9b0c7cc_6", "91e22800581543b5be4654f7b9b0c7cc_11", "91e22800581543b5be4654f7b9b0c7cc_8", "91e22800581543b5be4654f7b9b0c7cc_3", "91e22800581543b5be4654f7b9b0c7cc_17", [...] ] If I try to perform some operations on these buckets, I get an error: root@ceph1-40-10~# radosgw-admin bucket rm --bucket=91e22800581543b5be4654f7b9b0c7cc_15 --purge-objects 2020-07-28T12:40:05.392+0200 7f500024a080 -1 ERROR: unable to remove bucket(2) No such file or directory I'm able to change the owner and even rename it, but anything else is ending in the same error above. This also seems to break the buckets functionality in the dashboard. My question is: what are those buckets, where do they come from and what is stored in there? And why one is not able to perform any ops on them? Any ideas? Best, eugen

1 0

Re: please help me fix iSCSI Targets not available
by David Thuong 28 Jul '20

28 Jul '20

i user ceph octopus v15

1 0

HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error
by Fabrizio Cuseo 28 Jul '20

28 Jul '20

Hello, I use ceph with proxmox, release 14.2.9, with bluestore OSD. I had a problem in a replica 2 pool (i know, is dangerous), with an unexpected clone. I have deleted the RBD image that used the damaged object (with his snapshot), and now ceph can't trim and clean. How can I restore the clean and health status ? Regards, Fabrizio

2 1

Re: Cluster became unresponsive: e5 handle_auth_request failed to assign global_id
by Илья Борисович Волошин 28 Jul '20

28 Jul '20

No, they are stored locally on ESXi data storage on top of hardware RAID5 built with SAS/SATA (different hardware on hosts). Also, I've tried going back to the snapshot taken just after all monitors and OSDs were added to cluster. The host boots fine and is working as it should, however, after the next reboot this problem appears (no changes to configuration were made). And another thing - even if docker container for mgr is running and gives no errors in logs neither inside the container nor on the parent host the mgr doesn't bind to any ports it should: 6800, 6801 and 8443 for dashboard. Not sure if it is the reason or the consequence of this problem. вт, 28 июл. 2020 г. в 11:37, Anthony D'Atri <anthony.datri(a)gmail.com>: > Are your mon DBs on SSDs? > > > On Jul 27, 2020, at 7:28 AM, Илья Борисович Волошин < > i.voloshin(a)simplesolution.pro> wrote: > > > > Here are all the active ports on mon1 (with the exception of sshd and > ntpd): > > > > # netstat -npl > > Proto Recv-Q Send-Q Local Address Foreign Address State > > PID/Program name > > tcp 0 0 <mon1_ip>:3300 0.0.0.0:* LISTEN > > 1582/ceph-mon > > tcp 0 0 <mon1_ip>:6789 0.0.0.0:* > LISTEN > > 1582/ceph-mon > > tcp6 0 0 :::9093 :::* > LISTEN > > 908/alertmanager > > tcp6 0 0 :::9094 :::* > LISTEN > > 908/alertmanager > > tcp6 0 0 :::9095 :::* > LISTEN > > 896/prometheus > > tcp6 0 0 :::9100 :::* > LISTEN > > 906/node_exporter > > tcp6 0 0 :::3000 :::* > LISTEN > > 882/grafana-server > > udp6 0 0 :::9094 :::* > > 908/alertmanager > > > > I've tried telnet from mon1 host, can connect to 3300 and 6789: > > > > # telnet <mon1_ip> 3300 > > > Trying <mon1_ip>... > > Connected to <mon1_ip>. > > Escape character is '^]'. > > ceph v2 > > > > # telnet <mon1_ip> 6789 > > Trying <mon1_ip>... > > Connected to <mon1_ip>. > > Escape character is '^]'. > > ceph v027QQ > > > > 6800 and 6801 refuse connection: > > > > # telnet <mon1_ip> 6800 > > Trying <mon1_ip>... > > telnet: Unable to connect to remote host: Connection refused > > > > I don't see any errors in the log related to failures to bind... and all > > CEPH systemd services are running as far as I can tell: > > > > # systemctl list-units -a | grep ceph > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)alertmanager.mon1.service > > loaded active running Ceph > > alertmanager.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)crash.mon1.service > > loaded active running Ceph crash.mon1 > > for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)grafana.mon1.service > > loaded active running Ceph > grafana.mon1 > > for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)mgr.mon1.peevkl.service > > loaded active running Ceph > > mgr.mon1.peevkl for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)mon.mon1.service > > loaded active running Ceph mon.mon1 > for > > e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)node-exporter.mon1.service > > loaded active running Ceph > > node-exporter.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)prometheus.mon1.service > > loaded active running Ceph > > prometheus.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > system-ceph\x2de30397f0\x2dcc32\x2d11ea\x2d8c8e\x2d000c29469cd5.slice > > loaded active active > > system-ceph\x2de30397f0\x2dcc32\x2d11ea\x2d8c8e\x2d000c29469cd5.slice > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5.target > > loaded active active Ceph cluster > > e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph.target > > loaded active active All Ceph clusters > > and services > > > > Here are currently active docker images: > > > > # docker ps > > CONTAINER ID IMAGE COMMAND > > CREATED STATUS PORTS NAMES > > dfd8dbeccf1e ceph/ceph:v15 "/usr/bin/ceph-mgr -…" > > 41 minutes ago Up 41 minutes > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-mgr.mon1.peevkl > > 9452d1db7ffb ceph/ceph:v15 "/usr/bin/ceph-mon -…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-mon.mon1 > > 703ec4a43824 prom/prometheus:v2.18.1 "/bin/prometheus --c…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-prometheus.mon1 > > d816ec5e645f ceph/ceph:v15 "/usr/bin/ceph-crash…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-crash.mon1 > > 38d283ba6424 ceph/ceph-grafana:latest "/bin/sh -c 'grafana…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-grafana.mon1 > > cc119ec8f09a prom/node-exporter:v0.18.1 "/bin/node_exporter …" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-node-exporter.mon1 > > aa1d339c4100 prom/alertmanager:v0.20.0 "/bin/alertmanager -…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-alertmanager.mon1 > > > > iptables are active, I tried setting all chain policies to ACCEPT (didn't > > help), the rules are as such: > > > > 0 0 CEPH tcp -- * * 0.0.0.0/0 > > 0.0.0.0/0 tcp dpt:6789 > > 5060 303K CEPH tcp -- * * 0.0.0.0/0 > > 0.0.0.0/0 multiport dports 6800:7300 > > > > Chain CEPH includes addresses for monitors and OSDs. > > > > пн, 27 июл. 2020 г. в 17:07, Dino Godor <dg(a)terralink.de>: > > > >> Hi, > >> > >> have you tried to locally connect to the ports with netcat (or telnet)? > >> > >> Is the process listening ? (something like netstat -4ln or the current > >> equivalent thereof) > >> > >> Is the old (new) Firewall maybe still running ? > >> > >> > >> On 27.07.20 16:00, Илья Борисович Волошин wrote: > >>> Hello, > >>> > >>> I've created an Octopus 15.2.4 cluster with 3 monitors and 3 OSDs (6 > >> hosts > >>> in total, all ESXi VMs). It lived through a couple of reboots without > >>> problem, then I've reconfigured the main host a bit: > >>> set iptables-legacy as current option in update-alternatives (this is a > >>> Debian10 system), applied a basic ruleset of iptables and restarted > >> docker. > >>> > >>> After that the cluster became unresponsive (any ceph command hangs > >>> indefinitely). I can use admin socket to manipulate config though. > >> Setting > >>> debug_ms to 5 I see this in the logs (timestamps cut for readability): > >>> > >>> 7f4096f41700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> [v2:<mon2_ip>:3300/0,v1:<mon2_ip>:6789/0] conn(0x55c21b975800 > >>> 0x55c21ab45180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx= > >>> 0).send_message enqueueing message m=0x55c21bd84a00 type=67 > >> mon_probe(probe > >>> e30397f0-cc32-11ea-8c8e-000c29469cd5 name mon1 mon_release octopus) v7 > >>> 7f4098744700 1 -- >> > >>> [v2:<mon1_ip>:6800/561959008,v1:<mon1_ip>:6801/561959008] > >>> conn(0x55c21b974400 msgr2=0x55c21ab45600 unknown :-1 > >> s=STATE_CONNECTING_RE > >>> l=0).process reconnect failed to v2:81.200.2 > >>> .152:6800/561959008 > >>> 7f4098744700 2 -- >> > >>> [v2:<mon1_ip>:6800/561959008,v1:<mon1_ip>:6801/561959008] > >>> conn(0x55c21b974400 msgr2=0x55c21ab45600 unknown :-1 > >> s=STATE_CONNECTING_RE > >>> l=0).process connection refused! > >>> > >>> and this: > >>> > >>> 7f4098744700 2 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0)._fault on lossy channel, failing > >>> 7f4098744700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0).stop > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0).reset_recv_state > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0).reset_security > >>> 7f409373a700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 > >> rx=0 > >>> tx=0).accept > >>> 7f4098744700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=BANNER_ACCEPTING > pgs=0 > >>> cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=HELLO_ACCEPTING pgs=0 > >>> cs=0 l=0 rx=0 tx=0).handle_hello received hello: peer_type=8 > >>> peer_addr_for_me=v2:<mon1_ip>:3300/0 > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=HELLO_ACCEPTING pgs=0 > >>> cs=0 l=0 rx=0 tx=0).handle_hello getsockname says I am <mon1_ip>:3300 > >> when > >>> talking to v2:<mon1_ip>:49012/0 > >>> 7f4098744700 1 mon.mon1@0(probing) e5 handle_auth_request failed to > >> assign > >>> global_id > >>> > >>> Config (the result of ceph --admin-daemon > >>> /run/ceph/e30397f0-cc32-11ea-8c8e-000c29469cd5/ceph-mon.mon1.asok > config > >>> show): > >>> https://pastebin.com/kifMXs9H > >>> > >>> I can connect to ports 3300 and 6789 with telnet; 6800 and 6801 return > >>> 'process connection refused' > >>> > >>> Setting all iptables policies to ACCEPT didn't change anything. > >>> > >>> Where should I start digging to fix this problem? I'd like to at least > >>> understand why this happened before putting the cluster into > production. > >>> Any help is appreciated. > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Server error when trying to view this list in browser
by biohazd＠yahoo.com 28 Jul '20

28 Jul '20

Hi I often get "Server error - An error occurred while processing your request." when tryignt o view this lis in Fireofx, is it a known issues? https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/new

1 0

6 hosts fail cephadm check (15.2.4)
by Ml Ml 28 Jul '20

28 Jul '20

Hello, i get: [WRN] CEPHADM_HOST_CHECK_FAILED: 6 hosts fail cephadm check host ceph01 failed check: Failed to connect to ceph01 (ceph01). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@ceph01 host ceph02 failed check: Failed to connect to ceph02 (10.10.1.2). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@ceph02 host ceph03 failed check: Failed to connect to ceph03 (10.10.1.3). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@ceph03 host ceph04 failed check: Failed to connect to ceph04 (10.10.1.4). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@ceph04 host ceph05 failed check: Failed to connect to ceph05 (10.10.1.5). Check that the host is reachable and accepts connections using the cephadm SSH key you may want to run: > ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get mgr/cephadm/ssh_identity_key) root@ceph05 host ceph06 failed check: Failed to connect to ceph06 (10.10.1.6). Check that the host is reachable and accepts connections using the cephadm SSH key on ceph01 i run: ceph cephadm get-ssh-config > /tmp/ceph.conf ceph config-key get mgr/cephadm/ssh_identity_key > /tmp/ceph.key chmod 600 /tmp/ceph.key ssh -F /tmp/ceph.conf -i /tmp/ceph.key root@ceph01 (which works) So i can not understand the errors above. root@ceph01:~# ceph versions { "mon": { "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 3 }, "mgr": { "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 3 }, "osd": { "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 56 }, "mds": { "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 1 }, "overall": { "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 63 } } root@ceph01:~# dpkg -l |grep ceph ii ceph-base 15.2.4-1~bpo10+1 amd64 common ceph daemon libraries and management tools ii ceph-common 15.2.4-1~bpo10+1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-deploy 2.0.1 all Ceph-deploy is an easy to use configuration tool ii ceph-fuse 15.2.4-1~bpo10+1 amd64 FUSE-based client for the Ceph distributed file system ii ceph-grafana-dashboards 15.2.4-1~bpo10+1 all grafana dashboards for the ceph dashboard ii ceph-mds 15.2.4-1~bpo10+1 amd64 metadata server for the ceph distributed file system ii ceph-mgr 15.2.4-1~bpo10+1 amd64 manager for the ceph distributed storage system ii ceph-mgr-cephadm 15.2.4-1~bpo10+1 all cephadm orchestrator module for ceph-mgr ii ceph-mgr-dashboard 15.2.4-1~bpo10+1 all dashboard module for ceph-mgr ii ceph-mgr-diskprediction-cloud 15.2.4-1~bpo10+1 all diskprediction-cloud module for ceph-mgr ii ceph-mgr-diskprediction-local 15.2.4-1~bpo10+1 all diskprediction-local module for ceph-mgr ii ceph-mgr-k8sevents 15.2.4-1~bpo10+1 all kubernetes events module for ceph-mgr ii ceph-mgr-modules-core 15.2.4-1~bpo10+1 all ceph manager modules which are always enabled ii ceph-mon 15.2.4-1~bpo10+1 amd64 monitor server for the ceph storage system ii ceph-osd 15.2.4-1~bpo10+1 amd64 OSD server for the ceph storage system ii cephadm 15.2.4-1~bpo10+1 amd64 cephadm utility to bootstrap ceph daemons with systemd and containers ii libcephfs1 10.2.11-2 amd64 Ceph distributed file system client library ii libcephfs2 15.2.4-1~bpo10+1 amd64 Ceph distributed file system client library ii python-ceph-argparse 14.2.8-1 all Python 2 utility libraries for Ceph CLI ii python3-ceph-argparse 15.2.4-1~bpo10+1 all Python 3 utility libraries for Ceph CLI ii python3-ceph-common 15.2.4-1~bpo10+1 all Python 3 utility libraries for Ceph ii python3-cephfs 15.2.4-1~bpo10+1 amd64 Python 3 libraries for the Ceph libcephfs library root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_WARN 6 hosts fail cephadm check failed to probe daemons or devices 7 nearfull osd(s) Reduced data availability: 1 pg inactive Low space hindering backfill (add storage if this doesn't resolve itself): 26 pgs backfill_toofull Degraded data redundancy: 202495/33226941 objects degraded (0.609%), 26 pgs degraded, 26 pgs undersized 3 pool(s) nearfull services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 39m) mgr: ceph02(active, since 77m), standbys: ceph03, ceph01 mds: 2 up:standby osd: 61 osds: 56 up (since 41m), 55 in (since 41m); 27 remapped pgs data: pools: 3 pools, 2049 pgs objects: 11.08M objects, 37 TiB usage: 113 TiB used, 28 TiB / 141 TiB avail pgs: 0.049% pgs not active 202495/33226941 objects degraded (0.609%) 9238/33226941 objects misplaced (0.028%) 1025 active+clean 887 active+clean+snaptrim_wait 110 active+clean+snaptrim 25 active+undersized+degraded+remapped+backfill_toofull 1 undersized+degraded+remapped+backfill_toofull+peered 1 active+remapped+backfilling io: client: 1.0 KiB/s rd, 140 KiB/s wr, 0 op/s rd, 1 op/s wr recovery: 30 MiB/s, 8 objects/s I already restarted the mgr on ceph02. Thanks, Michael

2 1

Ceph pool at 90% capacity - rbd rm is timing out - any way to rescue?
by Victor Hooi 27 Jul '20

27 Jul '20

Hi, We've let our Ceph pool (Octopus) get into a bad state, with around 90% full: # ceph health > HEALTH_ERR 1/4 mons down, quorum > angussyd-kvm01,angussyd-kvm02,angussyd-kvm03; 3 backfillfull osd(s); 1 full > osd(s); 14 nearfull osd(s); Low space hindering backfill (add storage if > this doesn't resolve itself): 580 pgs backfill_toofull; Degraded data > redundancy: 1860769/9916650 objects degraded (18.764%), 597 pgs degraded, > 580 pgs undersized; 323 pgs not deep-scrubbed in time; 189 pgs not scrubbed > in time; Full OSDs blocking recovery: 17 pgs recovery_toofull; 4 pool(s) > full; 1 pools have too many placement groups At this point, even trying to run 'rbd rm" or "rbd du" seems to time out. (I am however, able to run "rbd ls -l" which shows me rbd image size - I assume that's before taking into account thin-provisioning). Is there any way to rescue this pool? Or at least some way to force delete some of the large images? Regards, Victor

1 0

Cluster became unresponsive: e5 handle_auth_request failed to assign global_id
by Илья Борисович Волошин 27 Jul '20

27 Jul '20

Hello, I've created an Octopus 15.2.4 cluster with 3 monitors and 3 OSDs (6 hosts in total, all ESXi VMs). It lived through a couple of reboots without problem, then I've reconfigured the main host a bit: set iptables-legacy as current option in update-alternatives (this is a Debian10 system), applied a basic ruleset of iptables and restarted docker. After that the cluster became unresponsive (any ceph command hangs indefinitely). I can use admin socket to manipulate config though. Setting debug_ms to 5 I see this in the logs (timestamps cut for readability): 7f4096f41700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> [v2:<mon2_ip>:3300/0,v1:<mon2_ip>:6789/0] conn(0x55c21b975800 0x55c21ab45180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx= 0).send_message enqueueing message m=0x55c21bd84a00 type=67 mon_probe(probe e30397f0-cc32-11ea-8c8e-000c29469cd5 name mon1 mon_release octopus) v7 7f4098744700 1 -- >> [v2:<mon1_ip>:6800/561959008,v1:<mon1_ip>:6801/561959008] conn(0x55c21b974400 msgr2=0x55c21ab45600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:81.200.2 .152:6800/561959008 7f4098744700 2 -- >> [v2:<mon1_ip>:6800/561959008,v1:<mon1_ip>:6801/561959008] conn(0x55c21b974400 msgr2=0x55c21ab45600 unknown :-1 s=STATE_CONNECTING_RE l=0).process connection refused! and this: 7f4098744700 2 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rx=0 tx=0)._fault on lossy channel, failing 7f4098744700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rx=0 tx=0).reset_recv_state 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 cs=0 l=1 rx=0 tx=0).reset_security 7f409373a700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).accept 7f4098744700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=BANNER_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=HELLO_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello received hello: peer_type=8 peer_addr_for_me=v2:<mon1_ip>:3300/0 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=HELLO_ACCEPTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello getsockname says I am <mon1_ip>:3300 when talking to v2:<mon1_ip>:49012/0 7f4098744700 1 mon.mon1@0(probing) e5 handle_auth_request failed to assign global_id Config (the result of ceph --admin-daemon /run/ceph/e30397f0-cc32-11ea-8c8e-000c29469cd5/ceph-mon.mon1.asok config show): https://pastebin.com/kifMXs9H I can connect to ports 3300 and 6789 with telnet; 6800 and 6801 return 'process connection refused' Setting all iptables policies to ACCEPT didn't change anything. Where should I start digging to fix this problem? I'd like to at least understand why this happened before putting the cluster into production. Any help is appreciated.

2 3

rbd-nbd stuck request
by Herbert Alexander Faleiros 27 Jul '20

27 Jul '20

Hi, is there any way to fix it instead a reboot? [128632.995249] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9540 seconds [128663.718993] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9570 seconds [128694.434774] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9600 seconds [128725.154515] block nbd0: Possible stuck request 00000000b14a04af: control (read@2097152,4096B). Runtime 9630 seconds # ceph -v ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable) # rbd-nbd list-mapped # # uname -r 5.4.52-050452-generic Thanks, -- Herbert

3 6

Fwd: BlueFS assertion ceph_assert(h->file->fnode.ino != 1)
by Aleksei Zakharov 27 Jul '20

27 Jul '20

2 1

Push config to all hosts
by Cem Zafer 27 Jul '20

27 Jul '20

Hi, What is the best method(s) to push ceph.conf to all hosts in octopus (15.x)? Thanks.

2 1

please help me fix iSCSI Targets not available
by David Thuong 27 Jul '20

27 Jul '20

iSCSI Targets not available Please consult the documentation on how to configure and enable the iSCSI Targets management functionality. Available information: There are no gateways defined any idea to enable it. tks so much

2 1

cache tier dirty status
by Budai Laszlo 27 Jul '20

27 Jul '20

Hello all, is there a way to interrogate a cache tier pool about the number of dirty objects/bytes that it contains? Thank you, Laszlo

1 0

Reinitialize rgw garbage collector
by Michael Bisig 27 Jul '20

27 Jul '20

Hi all, I have a question about the garbage collector within RGWs. We run Nautilus 14.2.8 and we have 32 garbage objects in the gc pool with totally 39 GB of garbage that needs to be processed. When we run, radosgw-admin gc process --include-all objects are processed but most of them won't be deleted. This can be checked by using --debug-rgw=5 in the command and stat the objects which are mentioned that they have been processed. Also the monitoring doesn't show that a huge amount of objects are deleted by the gc. So, I assume that it doesn't actually delete the objects. It might be due to a renewed time stamp? (not sure about this) Is there anybody who had similar issues with removing a large amount of garbage and is there a way to let the gc delete the objects? Most of the objects within the gc list are __multipart__ objects. Are they processed differently than single part objects? E.g. collect all the multiparts before the deletion actually happens or how is this implemented? The garbage is still increasing and the gc cannot process things what scares us a bit. Also, we cannot bypass the gc because the bucket is still in use. I also thought about reinitializing the GC in order to get an up to date list of garbage. (some entries show with `radosgw-admin gc list --include-all` are over a month old) Is there a way to make this happen and how save is it? I thought about exporting the omapobjects from the gc pool (as a backup) and delete the objects within the pool (or rename the pool). I appreciate any input and thank you in advance. Regards, Michael

1 0

Ceph-deploy on rhel.
by sathvik vutukuri 27 Jul '20

27 Jul '20

Hi, Did any one installed ceph-deploy on rhel7 with rados gate way. I see there are no rpms available for rhel7 in ceph-deploy in download.ceph.com for nautilis , luminous, octopus versions. Is ceph-deploy go to for rhel7???

3 9

large omap objects
by Zhenshi Zhou 27 Jul '20

27 Jul '20

Hi all, I have a cluster providing object storage. The cluster has worked well until someone saves flink checkpoints in the 'flink' bucket. I checked its behavior and I find that the flink saves the current checkpoint data and delete the former ones frequently. I suppose that it makes the bucket index get large omap object. I got '1 large omap objects' warning message these days. And after I check the cluster status and logs, of course, all of those large omap object points to exactly the same one index. The warning message: *cluster [WRN] Large omap object found. Object: 17:2f908b17:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.166289.1:head PG: 17.e8d109f4 (17.74) Key count: 568681 Size (bytes): 149443581* I did 'bilog trim' and 'pg deep-scrub' and the cluster became health again. However, I cannot do this all the time. Is there a way to solve this issue permanently? Thanks

1 0

OSD_SCRUB_ERRORS 1 scrub errors
by Abhimnyu Dhobale 25 Jul '20

25 Jul '20

Dear Team, Ceph is showing below errors frequently on different Disks. every time after pg repair it is resolving. # ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.574 is active+clean+inconsistent, acting [19,25,2] # cat /var/log/ceph/ceph-osd.19.log | grep error 2020-07-12 11:42:11.824 7f864e0b2700 -1 log_channel(cluster) log [ERR] : 1.574 shard 25 soid 1:2ea0a7a3:::rbd_data.515c96b8b4567.0000000000007a7c:head : candidate had a read error 2020-07-12 11:42:15.035 7f86520ba700 -1 log_channel(cluster) log [ERR] : 1.574 deep-scrub 1 errors # ceph --version ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable) # rados -p vpsacephcl01 list-inconsistent-obj 1.3c9 --format=json-pretty { "epoch": 845, "inconsistents": [ { "object": { "name": "rbd_data.515c96b8b4567.000000000000c377", "nspace": "", "locator": "", "snap": "head", "version": 21101 }, "errors": [], "union_shard_errors": [ "read_error" ], "selected_object_info": { "oid": { "oid": "rbd_data.515c96b8b4567.000000000000c377", "key": "", "snapid": -2, "hash": 867656649, "max": 0, "pool": 1, "namespace": "" }, "version": "853'21101", "prior_version": "853'21100", "last_reqid": "client.2317742.0:24909022", "user_version": 21101, "size": 4194304, "mtime": "2020-07-16 21:02:20.564245", "local_mtime": "2020-07-16 21:02:20.572003", "lost": 0, "flags": [ "dirty", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xffffffff", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 5, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 19, "primary": true, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x8ebd7de4" }, { "osd": 24, "primary": false, "errors": [ "read_error" ], "size": 4194304 } ] } ] } -- Thanks & Regards Abhimnyu Dhobale

3 2

journal based mirroring works but snapshot based not
by yveskretzschmar＠web.de 24 Jul '20

24 Jul '20

Hi, i have a working journal based mirror setup initially created with nautilus. I recently upgraded to octopus (15.2.4) to use snapshot based mirroring. After that I disabled mirroring for the first image and reenabled it snapshot based. The remote Image is created but it never starts syncing. Network usage is only about some kb/s. And after disabling mirroring on the image the remote images doesn't get deleted. The same is with a new test pool and new images. I also remove the peer from the pools and disabled mirroring on the pools and recreated it from scratch, because I've found this: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LRBDKQ2YXQN… bit it didn't resolved it. Mirroring daemon shows the follwing after enabling snapshot based mirroring on an image: 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images: peer_uuid=3d2164b5-96ca-4ad6-9fe3-4eb1f6c02f3e, added_count=1, removed_count=0 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images_added: peer_uuid=3d2164b5-96ca-4ad6-9fe3-4eb1f6c02f3e, global_image_ids=[d7f7d308-d18b-488f-9dcc-87f8e77f05cc] 2020-07-24T13:45:40.253+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 add_image: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 start_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=MAP_UPDATE 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 execute_policy_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, policy_action=MAP 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 map: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477 2020-07-24T13:45:41.254+0200 7f33da61c700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_image_mapping: updates=[{global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477}], removes=[] 2020-07-24T13:45:41.258+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 finish_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=MAP_UPDATE, r=0 2020-07-24T13:45:41.294+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:41.402+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:41.502+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:41.606+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 start_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=ACQUIRE 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 notify_listener_acquire_release_images: acquire=[{global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477}], release=[] 2020-07-24T13:45:42.258+0200 7f33da61c700 5 rbd::mirror::NamespaceReplayer: 0x55cfcc4c6000 handle_acquire_image: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, instance_id=10464477 2020-07-24T13:45:42.258+0200 7f33dae1d700 5 rbd::mirror::image_map::Policy: 0x55cfc8825a40 finish_action: global_image_id=d7f7d308-d18b-488f-9dcc-87f8e77f05cc, state=ASSOCIATING, action_type=ACQUIRE, r=0 2020-07-24T13:45:42.306+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images: peer_uuid=, added_count=1, removed_count=0 2020-07-24T13:45:42.306+0200 7f33dae1d700 5 rbd::mirror::ImageMap: 0x55cfcc4c0480 update_images_added: peer_uuid=, global_image_ids=[d7f7d308-d18b-488f-9dcc-87f8e77f05cc] 2020-07-24T13:45:42.330+0200 7f33a3daf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 init: failed to retrieve mirror peer uuid from remote pool 2020-07-24T13:45:42.330+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running 2020-07-24T13:45:46.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:46.402+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:46.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:46.606+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:51.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:51.406+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:51.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:51.610+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:45:52.330+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running 2020-07-24T13:45:56.298+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:45:56.406+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:45:56.506+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:45:56.610+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:46:01.303+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc7c1bd40 handle_acked: instance_ids=[10464466] 2020-07-24T13:46:01.407+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfc9424240 handle_acked: instance_ids=[10464470] 2020-07-24T13:46:01.511+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcac66240 handle_acked: instance_ids=[10464474] 2020-07-24T13:46:01.611+0200 7f33dae1d700 5 rbd::mirror::Instances: 0x55cfcc4c0360 handle_acked: instance_ids=[10464477] 2020-07-24T13:46:02.331+0200 7f33dae1d700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55cfcd28c800 get_replay_status: replay not running Regards, Yves

3 8

ceph orch apply [osd, mon] -i YAML file not found
by Hayashida, Mami 24 Jul '20

24 Jul '20

I am exploring the new cephadm tool by deploying a tiny test cluster using OpenStack VMs, but have encountered the following issue: ceph orch apply osd -i <yaml file> command fails with "No such file or directory: <yaml file>". (Monitor setup with a yaml file resulted in the same error) ``` root@octopus-mon-1:~# ls -lh total 12K drwxr-xr-x 2 root root 4.0K Jul 23 14:20 Playbooks -rw-r--r-- 1 root root 99 Jul 23 16:16 mon_spec.yml -rw-r--r-- 1 root root 127 Jul 23 16:03 osd_spec.yml root@octopus-mon-1:~# cat osd_spec.yml --- service_type: osd service_id: default_drive_group placement: host_pattern: 'octopus-osd-[1-3]' data_devices: all: true root@octopus-mon-1:~# ceph orch apply osd -i /root/osd_spec.yml --dry-run --verbose INFO:cephadm:Inferring fsid 9e87977c-cd16-11ea-b571-fa163ec9a1ad INFO:cephadm:Inferring config /var/lib/ceph/9e87977c-cd16-11ea-b571-fa163ec9a1ad/mon.octopus-mon-1/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name=None, cluster=None, cluster_timeout=None, completion=False, help=False, input_file='/root/osd_spec.yml', output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['orch', 'apply', 'osd', '--dry-run'] Can't open input file /root/osd_spec.yml: [Errno 2] No such file or directory: '/root/osd_spec.yml' root@octopus-mon-1:~# ceph orch --version INFO:cephadm:Inferring fsid 9e87977c-cd16-11ea-b571-fa163ec9a1ad INFO:cephadm:Inferring config /var/lib/ceph/9e87977c-cd16-11ea-b571-fa163ec9a1ad/mon.octopus-mon-1/config INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure

2 3

ceph-volume on FreeBSD: ImportError: cannot import name 'devices' from 'ceph_volume_zfs'
by Ronny Forberger 24 Jul '20

24 Jul '20

1 0

weight_set array in Ceph CRUSH
by Bobby 24 Jul '20

24 Jul '20

Hi, I have a confusion regarding a struct data type in Ceph CRUSH source code. Header fie in here https://github.com/ceph/ceph/blob/master/src/crush/crush.h If you see below. there is a struct data type namely _weight_set_. What I have understood going through different CRUSH maps, this _weight_set_ array should be a 2D array, am I right? struct crush_choose_arg { __s32 *ids; /*!< values to use instead of items */ __u32 ids_size; /*!< size of the __ids__ array */ struct crush_weight_set *weight_set; /*!< weight replacements for a given position */ __u32 weight_set_positions; }; BR Bobby

1 0

rbd map image with journaling
by Herbert Alexander Faleiros 24 Jul '20

24 Jul '20

Hi, is there any way to do that without disabling journaling? # rbd map image@snap rbd: sysfs write failed RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable image@snap journaling". In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (6) No such device or address # ceph -v ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable) # uname -r 5.4.52-050452-generic Thanks, -- Herbert

3 3

IPv6 connectivity gone for Ceph Telemetry
by Wido den Hollander 24 Jul '20

24 Jul '20

Hi, I was just checking on a few (13) IPv6-only Ceph clusters and I noticed that they couldn't send their Telemetry data anymore: telemetry.ceph.com has address 8.43.84.137 This server used to have Dual-Stack connectivity while it was still hosted at OVH. It seemed to have moved to Red Hat, but lost IPv6 connectivity. How can we get this back? Wido

2 1

Re: journal based mirroring works but snapshot based not
by Yves Kretzschmar-Schwipper 24 Jul '20

24 Jul '20

1 0

What affection to vm with Monitor down
by 展荣臻（信泰） 24 Jul '20

24 Jul '20

Hi,everyone Our production ceph cluster the version is hammer have 3 monitors and 300+ osds.Monitor daemon runs on host diffrent from osd daemon. openstack vm runs on rbd.Now we need to maintain monitor host so must shutdown monitor host a moment, one by one. But keep two monitor host online all the time. What is the impact of monitor down on vm? How to do reduce the impact?

2 1

how to ceph-objectstore-tool to copy OSD
by huxiaoyu＠horebdata.cn 24 Jul '20

24 Jul '20

Dear Cepher, On a Ceph node (Luminous 12.2.13), i had a HDD with smartctl inidicating going to fail soon but still operational, and i would like to replace it now. It is no fun to rebalance the data and then get a new disk in and do rebalanance again. I am thinging of using ceph-objectstore-tool to copy the data from the failing HDD to a new HDD on the same ceph node, with the same OSD ID. Is it possible, and how? thanks in advance for advice, samuel huxiaoyu(a)horebdata.cn

1 0

radosgw, public and private access on the same cluster ?
by Jean-Sebastien Landry 23 Jul '20

23 Jul '20

Hi everyone, we have a ceph cluster for object storage only, the rgws are accessible from the internet, and everything is ok. Now, one of our team/client required that their data should not ever be accessible from the internet. In any case of security bug/breach/whatever, they want to limit the access to their data from the local network. Before creating a second "private" cluster, is there a way to achieve this on our current "public" cluster? Is a multi-zone without replication would help me with that? A public rgws for public access on the "pub_zone", and a private rgws for private access on the "prv_zone"? pubzone.rgw.buckets.data prvzone.rgw.buckets.data If the "public" rgws is hacked, without the access_key/secret_key of the private zone, is there any possibilities to access the private zone? Does a multi-realms would help me to secure it more? Any input would be really appreciated. I don't want to put to much energy for false security and/or security by obscurity, so if these scenarios of multi-sites/multi-realms are useless, in a security point of view, please tell me. :-) Thanks! JS

2 2

Radosgw stuck in syncing status
by Nghia Viet Tran 23 Jul '20

23 Jul '20

Hi everyone, Our Ceph cluster is stuck in syncing status for a long time after executing the radosgw-admin data sync init command. ----- realm dcd64504-c445-4810-9b83-851875443bcd (storage) zonegroup 313a345a-4886-4cb3-8d06-0fe3919d591a (mastergroup) zone 76fc5fe2-9f89-4419-b611-ab275000b358 (dc01) metadata sync no sync (zone is master) data sync source: cc4e8e55-988a-430e-b1df-4d88f0c81f4f (dc02) syncing full sync: 117/128 shards full sync: 3 buckets to sync incremental sync: 11/128 shards data is behind on 117 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] ----- Can anyone give me a hint to let the synchronization job finish? Many thanks! -- Nghia Viet Tran (Mr) mgm technology partners Vietnam Co. Ltd 7 Phan Châu Trinh Đà Nẵng, Vietnam +84 935905659 nghia.viet.tran(a)mgm-tp.com<mailto:nghia.viet.tran@mgm-tp.com> www.mgm-tp.com<https://www.mgm-tp.com/en/> Visit us on LinkedIn<https://www.linkedin.com/company/mgm-technology-partners-vietnam-co-ltd> and Facebook<https://www.facebook.com/mgmTechnologyPartnersVietnam>! Innovation Implemented. General Director: Frank Müller Registered office: 7 Pasteur, Hải Châu 1, Hải Châu, Đà Nẵng MST/Tax 0401703955

4 3

activating cache tier while rbd is in use
by Budai Laszlo 23 Jul '20

23 Jul '20

Hello all, is it allowed to configure and activate a cache tier for a pool that contains used RBD images? The documentation (https://docs.ceph.com/docs/mimic/rados/operations/cache-tiering/) doesn't say anything about this, but we have experienced errors with our VMS consuming ceph rbd volumes. Is there any other preparation measure before we would issue the set-overlay command? Kind regards, Laszlo

1 0

Enabling Multi-MDS under Nautilus after cephfs-data-scan scan_links
by Wido den Hollander 22 Jul '20

22 Jul '20

Hi, I got involved in a case where a Nautilus cluster was experiencing MDSes asserting showing the backtrace mentioned in this ticket: https://tracker.ceph.com/issues/36349 ceph_assert(follows >= realm->get_newest_seq()); In the end we needed to use these tooling to get one MDS running again: https://docs.ceph.com/docs/master/cephfs/disaster-recovery-experts/#using-a… The root-cause seems to be that this Nautilus cluster was running Multi-MDS with a very high amount of CephFS snapshots. After a couple of days of scanning (scan_links seems single threaded!) we finally got a single MDS running again with a usable CephFS filesystem. At the moment chowns() are running to get all the permissions set back to what they should be. The question now outstanding: Is it safe to enable Multi-MDS again on a CephFS filesystem which still has these many snapshots and is running single at the moment? New snapshots are disabled at the moment, so those won't be created. In addition: How safe is it to remove snapshots? As this will result in metadata updates. Thanks Wido

1 0

Multisite error trim not working in Nautilus 14.2.8
by Amit Ghadge 22 Jul '20

22 Jul '20

Hi All, Sync error listed in error list and when I try to make empty/trim it's still not working, $ radosgw-admin sync error trim -AmitG

1 0

ceph mds recommended config
by mdyer＠mitre.org 22 Jul '20

22 Jul '20

Hello. I have a relatively new ceph installation that is only running ceph FS at the moment. We are seeing intermittent issues where "ceph -s" is reporting "MDS report slow requests" and sometimes the MDS crash and take a while to recover/replay or we have to manually restart an mds service to get the state back to HEALTH_OK. Is there any documentation for recommended configuration? Here is our cluster setup: 35 total nodes, 88 cores, 512GB ram, 100Gb network 2 ceph fs data pools, 1 is all ssd, the other is nvme 3 active MDS, 1 pinned to the nvme pool/dir, 1 pinned to another large directory, and the third has no pinning 2 standby MDS ceph config dump: mds advanced mds_beacon_grace 60.000000 mds basic mds_cache_memory_limit 68719476736 mds advanced mds_cache_trim_threshold 65536 mds advanced mds_recall_max_decay_rate 2.000000 Please let me know if more info is required. Thanks!

1 1

bluestore_prefer_deferred_size_hdd
by Frank Ritchie 22 Jul '20

22 Jul '20

Hi all, Is it safe to change bluestore_prefer_deferred_size_hdd for an OSD at runtime? thx Frank

2 2

Mimic is retired
by Abhishek Lekshmanan 22 Jul '20

22 Jul '20

This is to announce the retirement of v13.2.X Mimic stable release series, and there will no longer be any more backport releases to the Mimic series. Any more patches to the mimic branch will have to be tested by the developer submitting the patches and approved by the tech lead of the respective component before merge to keep the branch stable. The last release of Mimic was v13.2.10 released on Apr 2020. This is keeping up with the active 2 stable releases and 24 month support cycle, which is documented at https://docs.ceph.com/docs/master/releases/general/#lifetime-of-stable-rele… Users are requested to upgrade to Nautilus or Octopus. For the official blog post link please refer to https://ceph.io/releases/mimic-is-retired/ -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

S3 - Cannot rename/copy an encrypted object with sse-c-key inside S3 bucket.
by technical＠zylacomputing.com 22 Jul '20

22 Jul '20

Hi all, I am trying to move an encrypted file that was uploaded with sse-c-key inside an S3 bucket so i can rename the file in the same S3 bucket using aws cli version: 2.0.24, but i keep getting the error message shown below. This error message only happens when i am using move or copy on an encrypted object where the source and destination are the same bucket. When i copy an encrypted object from the bucket to a local machine and vice versa it works just fine. Command: aws s3 cp --endpoint=https://store-test-one.ddns.net --sse-c AES256 --sse-c-key 1234567890123456789012 3456789012 s3://one-disk/systemds7.txt s3://one-disk/systemds8.txt Error Message: copy failed: s3://one-disk/systemds7.txt to s3://one-disk/systemds8.txt An error occurred (NotImplemented) when calling the CopyObject operation: Unknown Anyone have any ideas why it fails when using move/copy on an encrypted object with sse-c-key where the source and destination is the same bucket? Thanks

2 1

Norton Setup
by paul21thomas9＠gmail.com 22 Jul '20

22 Jul '20

Norton Setup is the best antivirus software that allows you to protect your files as well as your system from various malware and viruses that are willing to attack your systems. Norton antivirus product is designed in a way to scan and eliminate any kind of possible threats such as Trojans, worms and viruses from affecting your personal computer, laptop, smartphone and tablet in the first place. Unlike other antivirus program, Norton holds onto a unique heuristics system to quickly identify the viruses that makes it stand apart as the best security service provider in the masses. You can purchase it from- https://w-wwnorton.com/setup/.

1 0

Resolve radosgw error list
by Nghia Viet Tran 22 Jul '20

22 Jul '20

Hi everyone, After trigger command radosgw-admin sync error list, Ceph returns the list of errors for each shards in the system e.g., { "shard_id": 30, "entries": [ { "id": "1_1592308047.968415_451.1", "section": "data", "name": "portal-images:76fc5fe2-9f89-4419-b611-ab275000b358.405220.1:8", "timestamp": "2020-06-16T11:47:27.968415Z", "info": { "source_zone": "30bae889-dc13-4957-a536-028394095356", "error_code": 5, "message": "failed to sync bucket instance: (5) Input/output error" } } ] } For further detail, I triggered the command radosgw-admin data sync status --shard-id=30 --source-zone=dc-02 { "shard_id": 30, "marker": { "status": "full-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" }, "pending_buckets": [], "recovering_buckets": [] } Seems like there is no problem in this shard so far… Can anyone help me on what I have to do to remove the errors above? Many thanks! -- Nghia Viet Tran (Mr) mgm technology partners Vietnam Co. Ltd 7 Phan Châu Trinh Đà Nẵng, Vietnam +84 935905659 nghia.viet.tran(a)mgm-tp.com<mailto:nghia.viet.tran@mgm-tp.com> www.mgm-tp.com<https://www.mgm-tp.com/en/> Visit us on LinkedIn<https://www.linkedin.com/company/mgm-technology-partners-vietnam-co-ltd> and Facebook<https://www.facebook.com/mgmTechnologyPartnersVietnam>! Innovation Implemented. General Director: Frank Müller Registered office: 7 Pasteur, Hải Châu 1, Hải Châu, Đà Nẵng MST/Tax 0401703955

1 0