August 2020 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

Remapped PGs
by David Orman 02 Mar '21

02 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

Default data pool in CEPH
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi, I have a CEPH 15.2.4 running in a docker. How to configure for use a specific data pool? i try put the follow line in the ceph.conf but the changes not working. . [client.myclient] rbd default data pool = Mydatapool I need it to configure for erasure pool with cloudstack Can help me? , where is the ceph conf we i need configure? Thanks. -- Untitled Document

4 6

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

v15.2.4 Octopus released
by David Galloway 17 Dec '20

17 Dec '20

We're happy to announce the fourth bugfix release in the Octopus series. In addition to a security fix in RGW, this release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-4-octopus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * Cephadm: There were a lot of small usability improvements and bug fixes: * Grafana when deployed by Cephadm now binds to all network interfaces. * `cephadm check-host` now prints all detected problems at once. * Cephadm now calls `ceph dashboard set-grafana-api-ssl-verify false` when generating an SSL certificate for Grafana. * The Alertmanager is now correctly pointed to the Ceph Dashboard * `cephadm adopt` now supports adopting an Alertmanager * `ceph orch ps` now supports filtering by service name * `ceph orch host ls` now marks hosts as offline, if they are not accessible. * Cephadm can now deploy NFS Ganesha services. For example, to deploy NFS with a service id of mynfs, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:: ceph orch apply nfs mynfs nfs-ganesha nfs-ns * Cephadm: `ceph orch ls --export` now returns all service specifications in yaml representation that is consumable by `ceph orch apply`. In addition, the commands `orch ps` and `orch ls` now support `--format yaml` and `--format json-pretty`. * Cephadm: `ceph orch apply osd` supports a `--preview` flag that prints a preview of the OSD specification before deploying OSDs. This makes it possible to verify that the specification is correct, before applying it. * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, and `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * RBD: The name of the rbd pool object that is used to store rbd trash purge schedule is changed from "rbd_trash_trash_purge_schedule" to "rbd_trash_purge_schedule". Users that have already started using `rbd trash purge schedule` functionality and have per pool or namespace schedules configured should copy "rbd_trash_trash_purge_schedule" object to "rbd_trash_purge_schedule" before the upgrade and remove "rbd_trash_purge_schedule" using the following commands in every RBD pool and namespace where a trash purge schedule was previously configured:: rados -p <pool-name> [-N namespace] cp rbd_trash_trash_purge_schedule rbd_trash_purge_schedule rados -p <pool-name> [-N namespace] rm rbd_trash_trash_purge_schedule or use any other convenient way to restore the schedule after the upgrade. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7447c15c6ff58d7fce91843b705a268a1917325c -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

4 5

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

block.db/block.wal device performance dropped after upgrade to 14.2.10
by Vladimir Prokofev 04 Dec '20

04 Dec '20

Good day, cephers! We've recently upgraded our cluster from 14.2.8 to 14.2.10 release, also performing full system packages upgrade(Ubuntu 18.04 LTS). After that performance significantly dropped, main reason beeing that journal SSDs are now have no merges, huge queues, and increased latency. There's a few screenshots in attachments. This is for an SSD journal that supports block.db/block.wal for 3 spinning OSDs, and it looks like this for all our SSD block.db/wal devices across all nodes. Any ideas what may cause that? Maybe I've missed something important in release notes?

6 20

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Advice on SSD choices for WAL/DB?
by Andrei Mikhailovsky 26 Nov '20

26 Nov '20

Hello, We are planning to perform a small upgrade to our cluster and slowly start adding 12TB SATA HDD drives. We need to accommodate for additional SSD WAL/DB requirements as well. Currently we are considering the following: HDD Drives - Seagate EXOS 12TB SSD Drives for WAL/DB - Intel D3 S4510 960GB or Intel D3 S4610 960GB Our cluster isn't hosting any IO intensive DBs nor IO hungry VMs such as Exchange, MSSQL, etc. From the documentation that I've read the recommended size for DB is between 1% and 4% of the size of the osd. Would 2% figure be sufficient enough (so around 240GB DB size for each 12TB osd?) Also, from your experience, which is a better model for the SSD DB/WAL? Would Intel S4510 be sufficient enough for our purpose or would the S4610 be a much better choice? Are there any other cost effective performance to consider instead of the above models? The same question to the HDD. Any other drives we should consider instead of the Seagate EXOS series? Thanks for you help and suggestions. Andrei

4 4

OSD memory leak?
by Frank Schilder 16 Nov '20

16 Nov '20

Hi all, on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load. What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats: Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } } Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Am I looking at a memory leak here or are these heap stats expected? I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 25

NoSuchKey on key that is visible in s3 list/radosgw bk
by Mariusz Gronczewski 06 Nov '20

06 Nov '20

Hi, I've got a problem on Octopus (15.2.3, debian packages) install, bucket S3 index shows a file: s3cmd ls s3://upvid/255/38355 --recursive 2020-07-27 17:48 50584342 s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 radosgw-admin bi list also shows it { "type": "plain", "idx": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "entry": { "name": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "instance": "", "ver": { "pool": 11, "epoch": 853842 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 50584342, "mtime": "2020-07-27T17:48:27.203008Z", "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", "storage_class": "", "owner": "filmweb-app", "owner_display_name": "filmweb app user", "content_type": "", "accounted_size": 50584342, "user_data": "", "appendable": "false" }, "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, but trying to download it via curl (I've set permissions to public0 only gets me <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>upvid</BucketName><RequestId>tx0000000000000000e716d-005f1f14cb-e478a-pl-war1</RequestId><HostId>e478a-pl-war1-pl</HostId></Error> (the actually nonexisting files shows access denied in same context) same with other tools: $ s3cmd get s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 /tmp download: 's3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' -> '/tmp/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' [1 of 1] ERROR: S3 error: 404 (NoSuchKey) cluster health is OK Any ideas what is happening here ? -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: admin(a)efigence.com

3 3

Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early
by Wido den Hollander 28 Sep '20

28 Sep '20

Hi, On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing OSDs randomly drop out of the cluster. Usually it's 2 to 4 OSDs spread out over different nodes. Each node has 16 OSDs and not all the failing OSDs are on the same node. The OSDs are marked as down and all they keep print in their logs: monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-06-04T07:57:17.706529-0400) Looking at their status through the admin socket: { "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", "whoami": 206, "state": "active", "oldest_map": 73697, "newest_map": 75795, "num_pgs": 19 } The message brought me to my own ticket I created 2 years ago: https://tracker.ceph.com/issues/23460 The first thing I've checked is NTP/time. Double, triple check this. All the times are in sync on the cluster. Nothing wrong there. Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. Restarting them brings them back right away and then within 24h some other OSDs will drop out. Has anybody seen this behavior with Octopus as well? Wido

2 1

Solve the issue of time scarcity via assignment help in Kuwait
by kevin wick 24 Sep '20

24 Sep '20

Are you facing a scarcity of timings or lack of time while composing assignments for Kuwait Universities? Are you not sure while taking the help of anybody to compose your assignments? For every student, assignments sound necessary but time-consuming tasks. You have to manage your time for your projects to score high marks as you can’t ignore your assignments during your study tenure. In this situation, if you can’t manage your time and require reliable assistance for your important task, place your order for assignment help even in Kuwait. Two important things that you need to keep in mind while working on your assignments are time management and quality content. Most students fail to submit their assignments on time because they could not manage their time and can’t collect relevant information for drafting their academic papers. However, don’t burst out and suffer your marks because of any reason. Instead of it, connect with assignment writing service provider and get your work done on time if you have less time to write your assignment. Many students achieve high marks on their projects due to the assistance of experts and professional writers. When you quote order for assignment writing help, you will provide enough time to engage yourself in some other academic tasks. If you don’t have time for writing your assignments and have no idea how to collect particulars for your work, transfer your project to professionals. Experts know how to arrange relevant information for composing the effective academic papers so you will not lose your marks. Expert’s knowledge and experience will allow you to connect with the right source of information and help you to score high marks. So, if you have issues in writing your academic papers, don’t forget to check out the services of online academic writing. https://www.greatassignmenthelp.com/kw/

3 2

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

Change crush rule on pool
by jesper＠krogh.cc 12 Sep '20

12 Sep '20

Hi I would like to change the crush rule so data lands on ssd instead of hdd, can this be done on the fly and migration will just happen or do I need to do something to move data? Jesper Sent from myMail for iOS

3 4

cephadm - How to deploy ceph cluster with a partition on SSD for block.db
by klemen＠psi-net.si 11 Sep '20

11 Sep '20

I'm trying to deploy a ceph cluster with a cephadm tool. I've already successfully done all steps except adding OSDs. My testing equipment consists of three hosts. Each host has SSD storage, where OS is installed into. On that storage I created partition, which can be used as a ceph block.db. Hosts have also 2 additional HDs (spinning drives) for OSD data. On docs I couldn't find how to deploy such configuration. Do you have any hints, how to do that? Thanks for help!

5 7

slow "rados ls"
by Marcel Kuiper 11 Sep '20

11 Sep '20

Hi One of my clusters running nautilus 14.2.8 is very slow (13 seconds or so where my other clusters are returning almost instantanious) when doing a 'rados --pool rc3-se.rgw.buckets.index ls' from one of the monitors. I checked - ceph status => OK - routing to/from osds ok (I see a lot of established connections to osds due to the command, nothing in syn_sent indicating incomplete handshake) - ping times are OK - no interface errors - no packet drops - no increasing send queus - and as far as I can see nothing out of the ordinary in mon and osd logs I have no clue how to debug the issue. If someone has pointers it would be much appreciated Kind Regards Marcel

4 16

Storage class usage stats
by Tobias Urdin 09 Sep '20

09 Sep '20

Hello, I've been trying to understand if there is any way to get usage information based on storage classes for buckets. Since there is no information available from the "radosgw-admin bucket stats" commands nor any other endpoint I tried to browse the source code but couldn't find any references where the storage class would be exposed in such a way. It also seems that RadosGW today is not saving any counters on amount of objects stored in storage classes when it's collecting usage stats, which means there is no such metadata saved for a bucket. I was hoping it was atleast saved but not exposed because then it would have been a easier fix than adding support to count number of objects in storage classes based on operations which would involve a lot of places and mean writing to the bucket metadata on each op :( Is my assumptions correct that there is no way to retrieve such information, meaning there is no way to measure such usage? If the answer is yes, I assume the only way to get something that could be measured would be to instead have multiple placement targets since that is exposed from in bucket info. The bad things would be though that you lose a lot of functionality related to lifecycle and moving a single object to another storage class. Best regards Tobias

2 2

Release mental stress using Hong Kong Assignment help
by james wick 08 Sep '20

08 Sep '20

Many students can’t complete their assignments within the assigned date because of some unavoidable circumstances in Hong Kong. It’s true that your mental stress raises hindrance and disturbs your concentration for your work. Because of stress and tension, you will not have enough thoughts to make your work effective. This situation directs you to the platform of Assignment Help services even in Hong Kong. This is because an unstable mind could not generate the right ideas to compose a worthy assignment. Make your assignment informative and productive using the online writing services of assignment experts. Professionals have a better understanding of the subject and know how to frame all information in the right format. So, you can use online assignment help when you have issues in composing your academic papers irrespective of any subject. https://www.greatassignmenthelp.com/hk/

4 3

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

Recover pgs from failed osds
by Vahideh Alinouri 07 Sep '20

07 Sep '20

Ceph cluster is updated from nautilus to octopus. On ceph-osd nodes we have high I/O wait. After increasing one of pool’s pg_num from 64 to 128 according to warning message (more objects per pg), this lead to high cpu load and ram usage on ceph-osd nodes and finally crashed the whole cluster. Three osds, one on each host, stuck at down state (osd.34 osd.35 osd.40). Starting the down osd service causes high ram usage and cpu load and ceph-osd node to crash until the osd service fails. The active mgr service on each mon host will crash after consuming almost all available ram on the physical hosts. I need to recover pgs and solving corruption. How can i recover unknown and down pgs? Is there any way to starting up failed osd? Below steps are done: 1- osd nodes’ kernel was upgraded to 5.4.2 before ceph cluster upgrading. Reverting to previous kernel 4.2.1 is tested for iowate decreasing, but it had no effect. 2- Recovering 11 pgs on failed osds by export them using ceph-objectstore-tools utility and import them on other osds. The result followed: 9 pgs are “down” and 2 pgs are “unknown”. 2-1) 9 pgs export and import successfully but status is “down” because of "peering_blocked_by" 3 failed osds. I cannot lost osds because of preventing unknown pgs from getting lost. pgs size in K and M. "peering_blocked_by": [ { "osd": 34, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 35, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 40, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] 2-2) 1 pg (2.39) export and import successfully, but after starting osd service (pg import to it), ceph-osd node RAM and CPU consumption increase and cause ceph-osd node to crash until the osd service fails. Other osds become "down" on ceph-osd node. pg status is “unknown”. I cannot use "force-create-pg" because of data lost. pg 2.39 size is 19G. # ceph pg map 2.39 osdmap e40347 pg 2.39 (2.39) -> up [32,37] acting [32,37] # ceph pg 2.39 query Error ENOENT: i don't have pgid 2.39 *pg 2.39 info on failed osd: # ceph-objectstore-tool --data-path /var/lib/ceph/osd/*ceph-34* --op info --pgid 2.39 { "pgid": "2.39", "last_update": "35344'6456084", "last_complete": "35344'6456084", "log_tail": "35344'6453182", "last_user_version": 10595821, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 146, "epoch_pool_created": 79, "last_epoch_started": 25208, "last_interval_started": 25207, "last_epoch_clean": 25208, "last_interval_clean": 25207, "last_epoch_split": 370, "last_epoch_marked_full": 0, "same_up_since": 8347, "same_interval_since": 25207, "same_primary_since": 8321, "last_scrub": "35328'6440139", "last_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "last_deep_scrub": "35261'6031075", "last_deep_scrub_stamp": "2020-08-17T01:59:26.606037+0430", "last_clean_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "prior_readable_until_ub": 0 }, "stats": { "version": "35344'6456082", "reported_seq": "11733156", "reported_epoch": "35344", "state": "active+clean", "last_fresh": "2020-08-19T14:16:18.587435+0430", "last_change": "2020-08-19T12:00:59.377747+0430", "last_active": "2020-08-19T14:16:18.587435+0430", "last_peered": "2020-08-19T14:16:18.587435+0430", "last_clean": "2020-08-19T14:16:18.587435+0430", "last_became_active": "2020-08-06T00:23:51.016769+0430", "last_became_peered": "2020-08-06T00:23:51.016769+0430", "last_unstale": "2020-08-19T14:16:18.587435+0430", "last_undegraded": "2020-08-19T14:16:18.587435+0430", "last_fullsized": "2020-08-19T14:16:18.587435+0430", "mapping_epoch": 8347, "log_start": "35344'6453182", "ondisk_log_start": "35344'6453182", "created": 146, "last_epoch_clean": 25208, "parent": "0.0", "parent_split_bits": 7, "last_scrub": "35328'6440139", "last_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "last_deep_scrub": "35261'6031075", "last_deep_scrub_stamp": "2020-08-17T01:59:26.606037+0430", "last_clean_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "log_size": 2900, "ondisk_log_size": 2900, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 19749578960, "num_objects": 2442, "num_object_clones": 20, "num_object_copies": 7326, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 2442, "num_whiteouts": 0, "num_read": 16120686, "num_read_kb": 82264126, "num_write": 19731882, "num_write_kb": 379030181, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 2861, "num_bytes_recovered": 21673259070, "num_keys_recovered": 32, "num_objects_omap": 2, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 152, "num_omap_keys": 16, "num_objects_repaired": 0 }, "up": [ 40, 35, 34 ], "acting": [ 40, 35, 34 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 40, "acting_primary": 40, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 25208, "hit_set_history": { "current_last_update": "0'0", "history": [] } } *pg 2.39 info on osd which import to it: # ceph-objectstore-tool --data-path /var/lib/ceph/osd/*ceph-37* --op info --pgid 2.39 PG '2.39' not found 2-3) 1 pg (2.79) is lost! This pg is not found on any of three failed osds (osd.34 osd.35 osd.40)! status is “unknown”. pg 2.79 export is failed: " PG '2.79' not found" # ceph pg map 2.79 Error ENOENT: i don't have pgid 2.79 # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-34 --op info --pgid 2.79 PG '2.79' not found 3- Using https://gitlab.lbader.de/kryptur/ceph-recovery/tree/master but it does not work for recent ceph versions and tested on “hammer” release. 4- Using https://ceph.io/planet/recovering-from-a-complete-node-failure/ but in lvm scenario I could not mount failed osd lv to new /var/lib/ceph/osd/ceph-x* .*Could not prepare and activate new osd to failed osd disk. 5- Setting pool min_size=1 that down pgs belong to it, restart osds that pgs import to them but no changes. 6- Seting pool min_size=1 that pg 2.39 belong to it, restart osds that pg import to them but no changes. 7- Repairing failed osds using ceph-objectstore-tools, making “in” and starting them but no changes. # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-x --op repair 8- Repairing 2 unknown pgs, but no changes. # ceph pg repaire 2.39 # ceph pg repair 2.79 9- Forcing recovery 2 unknown pgs, but no changes. # ceph pg force-recovery 2.39 # ceph pg force-recovery 2.79 10- Check PID count in ceph-osd nodes because of osd services failed to start. kernel.pid.max = 4194304 11- Raising osd_op_thread_suicide_timeout=900, but no change.

3 12

Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)
by VELARTIS Philipp Dürhammer 06 Sep '20

06 Sep '20

I have a productive 60 osd's cluster. No extra Journals. Its performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous. What should be possible 16 ssd's vs. 60 hhd's no extra journals?

6 16

Migrating Luminous → Nautilus "Required devices (data, and journal) not present for filestore"
by Stuart Longland 05 Sep '20

05 Sep '20

Hi all, I'm trying to migrate my 5-node cluster to Nautilus… Debian Stretch (I intend to update this once I'm onto Nautilus) as the OS, all 5 machines are combined OSD/MON/MDS nodes. Nodes were originally deployed with an earlier version of Ceph (can't recall now) using filestore on btrfs. Later, after an abortive attempt to move to Bluestore (it ran… veeeeerrryyy ssslllooowwwlllyyy), I moved back to filestore on xfs, which whilst no speed demon, worked well enough. Journal is on the OSD due to space constraints elsewhere in the nodes. So step 1: > If you are unsure whether or not your Luminous cluster has completed a full scrub of all PGs, you can check your cluster’s state by running: > > # ceph osd dump | grep ^flags > > In order to be able to proceed to Nautilus, your OSD map must include the recovery_deletes and purged_snapdirs flags. Tick to that, I see both flags. Step 2. > Make sure your cluster is stable and healthy (no down or recovering OSDs). (Optional, but recommended.) Tick, all healthy. Step 3. > Set the noout flag for the duration of the upgrade. (Optional, but recommended.): Done Step 4. > Upgrade monitors by installing the new packages and restarting the monitor daemons. Okay, so I update /etc/apt/sources.list.d/ceph.list to point to the `nautilus` repository, `apt-get update`, `apt-get dist-upgrade -y`. This goes without a hitch, I now have Ceph 12 binaries on my nodes. > systemctl restart ceph-mon.target Ran that, on all of them (and yes, they are all in the quorum)… > # ceph mon dump | grep min_mon_release > > should report: > > min_mon_release 14 (nautilus) I get this:> root@helium:~# ceph mon dump | grep min_mon_release > dumped monmap epoch 4 No `min_mon_release` mentioned anywhere. I tried re-starting the MON daemons on all 5 nodes, even doing it in parallel using Ansible… no dice. Nothing says what to do at this point. It's not reporting an earlier release, it's just not reporting full stop. Figuring that, well, I *did* update all the monitors, and re-start them, I press on. Step 5. > Upgrade ceph-mgr daemons by installing the new packages and restarting all manager daemons. Done, no issues. > Verify the ceph-mgr daemons are running by checking ceph -s: I get: > root@helium:~# ceph -s > cluster: > id: 45b532b7-aa3d-4754-9906-d4a70b57630c > health: HEALTH_WARN > noout flag(s) set > > services: > mon: 5 daemons, quorum hydrogen,helium,carbon,nitrogen,boron > mgr: hydrogen(active), standbys: carbon, helium > mds: cephfs-1/1/1 up {0=helium=up:active}, 2 up:standby > osd: 5 osds: 5 up, 5 in > flags noout > > data: > pools: 4 pools, 324 pgs > objects: 302.75k objects, 1.13TiB > usage: 3.52TiB used, 5.53TiB / 9.05TiB avail > pgs: 323 active+clean > 1 active+clean+scrubbing+deep > > io: > client: 45.0KiB/s rd, 1.10MiB/s wr, 1op/s rd, 38op/s wr So far so good. Step 6. > Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts: > systemctl restart ceph-osd.target I actually missed this step at first and went on to step 7, but doubled back and did it. Step 7. > If there are any OSDs in the cluster deployed with ceph-disk (e.g., almost any OSDs that were created before the Mimic release), you need to tell ceph-volume to adopt responsibility for starting the daemons. Okay, seems simple enough. Now I did this by mistake earlier, so I'll force it to ensure everything is up to scratch. > root@hydrogen:~# ceph-volume simple scan --force > stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device > stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. > Running command: /sbin/cryptsetup status /dev/sdc1 > --> OSD 5 got scanned and metadata persisted to file: /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json > --> To take over managment of this scanned OSD, and disable ceph-disk and udev, run: > --> ceph-volume simple activate 5 2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe > root@hydrogen:~# ceph-volume simple activate --all > --> activating OSD specified in /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json > --> Required devices (data, and journal) not present for filestore > --> filestore devices found: [u'data'] > --> RuntimeError: Unable to activate filestore OSD due to missing devices Uhh ohh? So, my journal is on the OSD itself: > root@hydrogen:~# ls -l /var/lib/ceph/osd/ceph-5/journal > -rw-r--r-- 1 ceph ceph 21474836480 Aug 29 14:16 /var/lib/ceph/osd/ceph-5/journal > root@hydrogen:~# cat /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json > { > "active": "ok", > "ceph_fsid": "45b532b7-aa3d-4754-9906-d4a70b57630c", > "cluster_name": "ceph", > "data": { > "path": "/dev/sdc1", > "uuid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe" > }, > "fsid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe", > "keyring": "…censored…", > "magic": "ceph osd volume v026", > "ready": "ready", > "require_osd_release": "", > "systemd": "", > "type": "filestore", > "whoami": 5 > } How do I tell it that the journal is on `/dev/sdc1` as a file within the XFS file store? -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.

1 2

Ceph Filesystem recovery with intact pools
by cyclic3.git＠gmail.com 02 Sep '20

02 Sep '20

Hi, I've had a complete monitor failure, which I have recovered from with the steps here: https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/… The data and metadata pools are there and are completely intact, but ceph is reporting that there are no filesystems, where (before the failure) there was one. Is there any way of putting the filesystem back together again without having to resort to having to rebuild a new metadata pool with cephfs-data-scan? I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Thanks, Harlan

4 11

osd regularly wrongly marked down
by Francois Legrand 01 Sep '20

01 Sep '20

Hi all, We have a ceph cluster in production with 6 osds servers (with 16x8TB disks), 3 mons/mgrs and 3 mdss. Both public and cluster networks are in 10GB and works well. After a major crash in april, we turned the option bluefs_buffered_io to false to workaround the large write bug when bluefs_buffered_io was true (we were in version 14.2.8 and the default value at this time was true). Since that time, we regularly have some osds wrongly marked down by the cluster after heartbeat timeout (heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15). Generally the osd restart and the cluster is back healthy, but several time, after many of these kick-off the osd reach the osd_op_thread_suicide_timeout and goes down definitely. We increased the osd_op_thread_timeout and osd_op_thread_suicide_timeout... The problems still occurs (but less frequently). Few days ago, we upgraded to 14.2.11 and revert the timeout to their default value, hoping that it will solve the problem (we thought that it should be related to this bug https://tracker.ceph.com/issues/45943) but it didn't. We still have some osds wrongly marked down. Can somebody help us to fix this problem ? Thanks. Here is an extract of an osd log at failure time: --------------------------------- 2020-08-28 02:19:05.019 7f03f1384700 0 log_channel(cluster) log [DBG] : 44.7d scrub starts 2020-08-28 02:19:25.755 7f040e43d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:19:25.755 7f040dc3c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 this last line is repeated more than 1000 times ... 2020-08-28 02:20:17.484 7f040d43b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:20:17.551 7f03f1384700 0 bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow operation observed for _collection_list, latency = 67.3532s, lat = 67s cid =44.7d_head start GHMAX end GHMAX max 25 ... 2020-08-28 02:20:22.600 7f040dc3c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:21:20.774 7f03f1384700 0 bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow operation observed for _collection_list, latency = 63.223s, lat = 63s cid =44.7d_head start #44:beffc78d:::rbd_data.1e48e8ab988992.00000000000011bd:0# end #MAX# max 2147483647 2020-08-28 02:21:20.774 7f03f1384700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:21:20.805 7f03f1384700 0 log_channel(cluster) log [DBG] : 44.7d scrub ok 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.16 down, but it is still running 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) log [DBG] : map e609411 wrongly marked me down at e609410 2020-08-28 02:21:21.099 7f03fd997700 1 osd.16 609411 start_waiting_for_healthy 2020-08-28 02:21:21.119 7f03fd997700 1 osd.16 609411 start_boot 2020-08-28 02:21:21.124 7f03f0b83700 1 osd.16 pg_epoch: 609410 pg[36.3d0( v 609409'481293 (449368'478292,609409'481293] local-lis/les=609403/609404 n=154651 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609410/609410/608752) [25,72] r=-1 lpr=609410 pi=[609403,609410)/1 luod=0'0 lua=609392'481198 crt=609409'481293 lcod 609409'481292 active mbc={}] start_peering_interval up [25,72,16] -> [25,72], acting [25,72,16] -> [25,72], acting_primary 25 -> 25, up_primary 25 -> 25, role 2 -> -1, features acting 4611087854031667199 upacting 4611087854031667199 ... 2020-08-28 02:21:21.166 7f03f0b83700 1 osd.16 pg_epoch: 609411 pg[36.56( v 609409'480511 (449368'477424,609409'480511] local-lis/les=609403/609404 n=153854 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609410/609410/609410) [103,102] r=-1 lpr=609410 pi=[609403,609410)/1 crt=609409'480511 lcod 609409'480510 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity public network em1 numa node 0 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity cluster network em2 numa node 0 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity objectstore and network numa nodes do not match 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity not setting numa affinity 2020-08-28 02:21:21.566 7f040a435700 1 osd.16 609413 tick checking mon for new map 2020-08-28 02:21:22.515 7f03fd997700 1 osd.16 609414 state: booting -> active 2020-08-28 02:21:22.515 7f03f0382700 1 osd.16 pg_epoch: 609414 pg[36.20( v 609409'483167 (449368'480117,609409'483167] local-lis/les=609403/609404 n=155171 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609361) [97,16,72] r=1 lpr=609414 pi=[609403,609414)/1 crt=609409'483167 lcod 609409'483166 unknown NOTIFY mbc={}] start_peering_interval up [97,72] -> [97,16,72], acting [97,72] -> [97,16,72], acting_primary 97 -> 97, up_primary 97 -> 97, role -1 -> 1, features acting 4611087854031667199 upacting 4611087854031667199 ... 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) [16,34,21] r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod 609409'479795 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [34,21] -> [16,34,21], acting [34,21] -> [16,34,21], acting_primary 34 -> 16, up_primary 34 -> 16, role -1 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) [16,34,21] r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod 609409'479795 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-08-28 02:21:24.738 7f03f1384700 0 log_channel(cluster) log [DBG] : 36.2f3 scrub starts 2020-08-28 02:22:18.857 7f03f1384700 0 log_channel(cluster) log [DBG] : 36.2f3 scrub ok

3 4

RandomCrashes on OSDs Attached to Mon Hosts with Octopus
by Denis Krienbühl 01 Sep '20

01 Sep '20

Hi! We've recently upgraded all our clusters from Mimic to Octopus (15.2.4). Since then, our largest cluster is experiencing random crashes on OSDs attached to the mon hosts. This is the crash we are seeing (cut for brevity, see links in post scriptum): { "ceph_version": "15.2.4", "utsname_release": "4.15.0-72-generic", "assert_condition": "r == 0", "assert_func": "void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)", "assert_file": "/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>", "assert_line": 11430, "assert_thread_name": "bstore_kv_sync", "assert_msg": "/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>: In function 'void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)' thread 7fc56311a700 time 2020-08-26T08:52:24.917083+0200\n/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>: 11430: FAILED ceph_assert(r == 0)\n", "backtrace": [ "(()+0x12890) [0x7fc576875890]", "(gsignal()+0xc7) [0x7fc575527e97]", "(abort()+0x141) [0x7fc575529801]", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a5) [0x559ef9ae97b5]", "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x559ef9ae993f]", "(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x3a0) [0x559efa0245b0]", "(BlueStore::_kv_sync_thread()+0xbdd) [0x559efa07745d]", "(BlueStore::KVSyncThread::entry()+0xd) [0x559efa09cd3d]", "(()+0x76db) [0x7fc57686a6db]", "(clone()+0x3f) [0x7fc57560a88f]" ] } Right before the crash occurs, we see the following message in the crash log: -3> 2020-08-26T08:52:24.787+0200 7fc569b2d700 2 rocksdb: [db/db_impl_compaction_flush.cc:2212 <http://db_impl_compaction_flush.cc:2212/>] Waiting after background compaction error: Corruption: block checksum mismatch: expected 2548200440, got 2324967102 in db/815839.sst offset 67107066 size 3808, Accumulated background error counts: 1 -2> 2020-08-26T08:52:24.852+0200 7fc56311a700 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: expected 2548200440, got 2324967102 in db/815839.sst offset 67107066 size 3808 code = 2 Rocksdb transaction: In short, we see a Rocksdb corruption error after background compaction, when this happens. When an OSD crashes, which happens about 10-15 times a day, it restarts and resumes work without any further problems. We are pretty confident that this is not a hardware issue, due to the following facts: * The crashes occur on 5 different hosts over 3 different racks. * There is no smartctl/dmesg output that could explain it. * It usually happens to a different OSD that did not crash before. Still we checked the following on a few OSDs/hosts: * We can do a manual compaction, both offline and online. * We successfully ran "ceph-bluestore-tool fsck --deep yes" on one of the OSDs. * We manually compacted a number of OSDs, one of which crashed hours later. The only thing we have noticed so far: It only happens to OSDs that are attached to a mon host. *None* of the non-mon host OSDs have had a crash! Does anyone have a hint what could be causing this? We currently have no good theory that could explain this, much less have a fix or workaround. Any help would be greatly appreciated. Denis Crash: https://public-resources.objects.lpg.cloudscale.ch/osd-crash/meta.txt <https://public-resources.objects.lpg.cloudscale.ch/osd-crash/meta.txt> Log: https://public-resources.objects.lpg.cloudscale.ch/osd-crash/log.txt <https://public-resources.objects.lpg.cloudscale.ch/osd-crash/log.txt>

7 11

rgw.none vs quota
by Jean-Sebastien Landry 01 Sep '20

01 Sep '20

Hi everyone, a bucket was overquota, (default quota of 300k objects per bucket), I enabled the object quota for this bucket and set a quota of 600k objects. We are on Luminous (12.2.12) and dynamic resharding is disabled, I manually do the resharding from 3 to 6 shards. Since then, radosgw-admin bucket stats report a `rgw.none` in the usage section for this bucket. I search the mailing-lists, bugzilla, github, it's look like I can ignore the rgw.none stats. (0 byte object, entry left in the index marked as cancelled...) but, the num_object in rgw.none is part of the quota usage. I bump the quota to 800k object to workaround the problem. (without resharding) Is there a way I can garbage collect the rgw.none? Is this problem fixed in Mimic/Nautilus/Octopus? "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 417827 }, "rgw.main": { "size": 1390778138502, "size_actual": 1391581007872, "size_utilized": 1390778138502, "size_kb": 1358181776, "size_kb_actual": 1358965828, "size_kb_utilized": 1358181776, "num_objects": 305637 } }, Thanks!

3 2

setting bucket quota using admin API does not work
by Youzhong Yang 01 Sep '20

01 Sep '20

Hi all, I tried to set bucket quota using admin API as shown below: admin/user?quota&uid=bse&bucket=test&quota-type=bucket with payload in json format: { "enabled": true, "max_size": 1099511627776, "max_size_kb": 1073741824, "max_objects": -1 } it returned success but the quota change did not happen, as confirmed by 'radosgw-admin bucket stats --bucket=test' command. Am I missing something obvious? Please kindly advise/suggest. By the way, I am using ceph mimic (v13.2.4). Setting quota by radosgw-admin quota set --bucket=${BUCK} --max-size=1T --quota-scope=bucket works, but I want to do it programmatically. Thanks in advance, -Youzhong

1 1

Cluster degraded after adding OSDs to increase capacity
by Dallas Jones 31 Aug '20

31 Aug '20

My 3-node Ceph cluster (14.2.4) has been running fine for months. However, my data pool became close to full a couple of weeks ago, so I added 12 new OSDs, roughly doubling the capacity of the cluster. However, the pool size has not changed, and the health of the cluster has changed for the worse. The dashboard shows the following cluster status: - PG_DEGRADED_FULL: Degraded data redundancy (low space): 2 pgs backfill_toofull - POOL_NEARFULL: 6 pool(s) nearfull - OSD_NEARFULL: 1 nearfull osd(s) Output from ceph -s: cluster: id: e5a47160-a302-462a-8fa4-1e533e1edd4e health: HEALTH_ERR 1 nearfull osd(s) 6 pool(s) nearfull Degraded data redundancy (low space): 2 pgs backfill_toofull services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 5w) mgr: ceph01(active, since 4w), standbys: ceph03, ceph02 mds: cephfs:1 {0=ceph01=up:active} 2 up:standby osd: 33 osds: 33 up (since 43h), 33 in (since 43h); 1094 remapped pgs rgw: 3 daemons active (ceph01, ceph02, ceph03) data: pools: 6 pools, 1632 pgs objects: 134.50M objects, 7.8 TiB usage: 42 TiB used, 81 TiB / 123 TiB avail pgs: 213786007/403501920 objects misplaced (52.983%) 1088 active+remapped+backfill_wait 538 active+clean 4 active+remapped+backfilling 2 active+remapped+backfill_wait+backfill_toofull io: recovery: 477 KiB/s, 330 keys/s, 29 objects/s Can someone steer me in the right direction for how to get my cluster healthy again? Thanks in advance! -Dallas

5 10

MDS troubleshooting documentation: ceph daemon mds.<name> dump cache
by Stefan Kooman 31 Aug '20

31 Aug '20

Hi list, We had some stuck ops on our MDS. In order to figure out why, we looked up the documention. The first thing it mentions is the following: ceph daemon mds.<name> dump cache /tmp/dump.txt Our MDS had 170 GB in cache at that moment. Turns out that is a sure way to get your active MDS replaced by a standby. Is this supposed to work on MDS with large cache size? If not, than a big warning sign to prohibit running this on MDSes with large caches would be appropriate. Gr. Stefan P.s. I think our only option was to get the active restarted at that point, but still.

2 1

Xfs kernel panic during rbd mount
by Shain Miley 31 Aug '20

31 Aug '20

Hi, A few weeks ago several of our rdb images became unresponsive after a few of our OSDs reached a near full state. Another member of the team rebooted the server that the rbd images are mounted on in an attempt to resolve the issue. In the meantime I added several more nodes to the cluster in order to get additional space. Here are some cluster details: root@rbd1:/var/log# ceph -v ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) root@rbd1:/var/log# uname -va Linux rbd1 4.15.0-48-generic #51~16.04.1-Ubuntu SMP Fri Apr 5 12:01:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux root@rbd1:/var/log# ceph -s cluster: id: 504b5794-34bd-44e7-a8c3-0494cf800c23 health: HEALTH_ERR crush map has legacy tunables (require argonaut, min is firefly) full ratio(s) out of order 2091546/274437905 objects misplaced (0.762%) Reduced data availability: 114 pgs inactive 1 slow requests are blocked > 32 sec. Implicated osds 152 4 stuck requests are blocked > 4096 sec. Implicated osds 6,99 services: mon: 3 daemons, quorum hqceph1,hqceph2,hqceph3 mgr: hqceph2(active), standbys: hqceph3 osd: 291 osds: 283 up, 265 in; 116 remapped pgs rgw: 1 daemon active data: pools: 17 pools, 8199 pgs objects: 91.48M objects, 292TiB usage: 880TiB used, 758TiB / 1.60PiB avail pgs: 1.390% pgs not active 2091546/274437905 objects misplaced (0.762%) 8040 active+clean 114 activating+remapped 40 active+clean+scrubbing+deep 3 active+clean+scrubbing 2 active+remapped+backfilling io: recovery: 41.4MiB/s, 12objects/s This morning I got on the server in order to map and mount the 5 or so RBD images that are shared out via samba on this server. After waiting about 10 minutes it was clear something was not 100% correct…here is what I found in the kern.log file: Aug 31 11:43:16 rbd1 kernel: [2158818.570948] libceph: mon0 10.35.1.201:6789 session established Aug 31 11:43:16 rbd1 kernel: [2158818.576306] libceph: client182797617 fsid 504b5794-34bd-44e7-a8c3-0494cf800c23 Aug 31 11:43:16 rbd1 kernel: [2158818.710199] rbd: rbd0: capacity 54975581388800 features 0x0 Aug 31 11:43:22 rbd1 kernel: [2158824.685353] XFS (rbd0): Mounting V4 Filesystem Aug 31 11:44:19 rbd1 kernel: [2158881.536794] XFS (rbd0): Starting recovery (logdev: internal) Aug 31 11:47:06 rbd1 kernel: [2159048.202835] INFO: task mount:33177 blocked for more than 120 seconds. Aug 31 11:47:06 rbd1 kernel: [2159048.203053] Not tainted 4.15.0-48-generic #51~16.04.1-Ubuntu Aug 31 11:47:06 rbd1 kernel: [2159048.203260] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 31 11:47:06 rbd1 kernel: [2159048.203523] mount D 0 33177 33011 0x00000000 Aug 31 11:47:06 rbd1 kernel: [2159048.203527] Call Trace: Aug 31 11:47:06 rbd1 kernel: [2159048.203538] __schedule+0x3d6/0x8b0 Aug 31 11:47:06 rbd1 kernel: [2159048.203542] ? __switch_to_asm+0x34/0x70 Aug 31 11:47:06 rbd1 kernel: [2159048.203546] schedule+0x36/0x80 Aug 31 11:47:06 rbd1 kernel: [2159048.203551] schedule_timeout+0x1db/0x370 Aug 31 11:47:06 rbd1 kernel: [2159048.203629] ? xfs_trans_read_buf_map+0xf8/0x330 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.203634] wait_for_completion+0xb4/0x140 Aug 31 11:47:06 rbd1 kernel: [2159048.203637] ? wake_up_q+0x70/0x70 Aug 31 11:47:06 rbd1 kernel: [2159048.203691] ? xfs_trans_read_buf_map+0xf8/0x330 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.203740] ? _xfs_buf_read+0x23/0x30 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.203787] xfs_buf_submit_wait+0x7f/0x220 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.203839] ? xfs_trans_read_buf_map+0xf8/0x330 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.203887] _xfs_buf_read+0x23/0x30 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.203933] xfs_buf_read_map+0x10a/0x190 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.203985] xfs_trans_read_buf_map+0xf8/0x330 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204022] xfs_read_agf+0x90/0x120 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204058] xfs_alloc_read_agf+0x49/0x1d0 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204094] xfs_alloc_pagf_init+0x29/0x60 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204141] xfs_initialize_perag_data+0x99/0x110 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204193] xfs_mountfs+0x79b/0x950 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204243] ? xfs_mru_cache_create+0x12b/0x170 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204294] xfs_fs_fill_super+0x428/0x5e0 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204300] mount_bdev+0x246/0x290 Aug 31 11:47:06 rbd1 kernel: [2159048.204349] ? xfs_test_remount_options.isra.16+0x60/0x60 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204398] xfs_fs_mount+0x15/0x20 [xfs] Aug 31 11:47:06 rbd1 kernel: [2159048.204402] mount_fs+0x3d/0x150 Aug 31 11:47:06 rbd1 kernel: [2159048.204406] vfs_kern_mount+0x67/0x110 Aug 31 11:47:06 rbd1 kernel: [2159048.204410] do_mount+0x201/0xd00 Aug 31 11:47:06 rbd1 kernel: [2159048.204414] ? __check_object_size+0xfc/0x1a0 Aug 31 11:47:06 rbd1 kernel: [2159048.204418] SyS_mount+0x95/0xe0 Aug 31 11:47:06 rbd1 kernel: [2159048.204423] do_syscall_64+0x73/0x130 Aug 31 11:47:06 rbd1 kernel: [2159048.204426] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Aug 31 11:47:06 rbd1 kernel: [2159048.204429] RIP: 0033:0x7f77a9a4fc5a Aug 31 11:47:06 rbd1 kernel: [2159048.204431] RSP: 002b:00007ffda3e83818 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 Aug 31 11:47:06 rbd1 kernel: [2159048.204435] RAX: ffffffffffffffda RBX: 0000000002424030 RCX: 00007f77a9a4fc5a Aug 31 11:47:06 rbd1 kernel: [2159048.204436] RDX: 0000000002424210 RSI: 0000000002424310 RDI: 0000000002424230 Aug 31 11:47:06 rbd1 kernel: [2159048.204438] RBP: 0000000000000000 R08: 0000000002424280 R09: 0000000000000012 Aug 31 11:47:06 rbd1 kernel: [2159048.204440] R10: 00000000c0ed0c00 R11: 0000000000000206 R12: 0000000002424230 Aug 31 11:47:06 rbd1 kernel: [2159048.204441] R13: 0000000002424210 R14: 0000000000000000 R15: 0000000000000003 Any suggestions on what I can/should do next? Thanks in advance, Shain Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley(a)npr.org

2 2

Bluestore does not defer writes
by Dennis Benndorf 31 Aug '20

31 Aug '20

Hi, today I recognized bad performance in our cluster. Running "watch ceph osd perf |sort -hk 2 -r" I found that all bluestore OSDs are slow on commit and that the commit timings are equal to their apply timings: For example Every 2.0s: ceph osd perf |sort -hk 2 -r 440 82 82 430 58 58 435 56 56 449 53 53 442 40 40 441 30 30 439 27 27 99 0 1 98 0 0 97 0 2 96 0 6 95 0 2 94 0 6 93 0 13 The once with zero commit timings are filestore and the others are bluestore osds. I did not see this after installing the new bluestore osds (maybe this occured later). Both types of osds have nvmes as journal/db. Servers have equal cpus/ram etc. The only tuning regarding bluestore is: bluestore_block_db_size = 69793218560 bluestore_prefer_deferred_size_hdd = 524288 In order to make a filestore like behavior, but that does not seem to work. Any tips? Regards Dennis

2 3

Persistent problem with slow metadata
by Momčilo Medić 31 Aug '20

31 Aug '20

Hi friends, Since deployment of our Ceph cluster we've been plagued by slow metadata error. Namely, cluster goes into HEALTH_WARN with a message similar to this one: 2 MDSs report slow metadata IOs 1 MDSs report slow requests 1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4] have slow ops. Here is a brief overview of our setup: - 7 OSD nodes with 6 OSD drives each - three of those are also monitors, managers and MDS - there is a single Ceph client (at the moment) - there is only CephFS being used (at the moment) - metadata for CephFS is on HDD (was on HDD, but we moved it as suggested - no improvement) Our expectation is that this is not a RAM issue as we have 64GiB of memory and is never fully utilized. It might be a CPU problem, as issue happens mostly during high loads (load of ~12 on a 8-core Intel Xeon Bronze 3106). However, the load is present on all OSD nodes, not just MDS ones. Cluster is used for (mostly nightly) backups and has no critical performance requirement. Interestingly, significant load across all nodes appears when running cleanup of outdated backups. This boils down to mostly truncating files and some removal, but it is usually small number of large files. Bellow you can find an example of "dump_ops_in_flight" output during the problem (which you may find useful - I couldn't make sense out of it). Should we invest into more powerfull CPU hardware (or should we move MDS roles to more powerful nodes)? Please let me know if I can share any more information to help resolve this thing. Thanks in advance! Kind regards, Momo. === { "ops": [ { "description": "client_request(client.22661659:706483006 create #0x10000002742/a-random-file 2020-08-23T23:09:33.919740+0200 caller_uid=117, caller_gid=121{})", "initiated_at": "2020-08-23T23:09:33.926509+0200", "age": 30.193027896, "duration": 30.193083934000001, "type_data": { "flag_point": "failed to authpin, subtree is being exported", "reqid": "client.22661659:706483006", "op_type": "client_request", "client_info": { "client": "client.22661659", "tid": 706483006 }, "events": [ { "time": "2020-08-23T23:09:33.926509+0200", "event": "initiated" }, { "time": "2020-08-23T23:09:33.926510+0200", "event": "throttled" }, { "time": "2020-08-23T23:09:33.926509+0200", "event": "header_read" }, { "time": "2020-08-23T23:09:33.926516+0200", "event": "all_read" }, { "time": "2020-08-23T23:09:33.926540+0200", "event": "dispatched" }, { "time": "2020-08-23T23:09:33.926595+0200", "event": "failed to authpin, subtree is being exported" } ] } } ], "num_ops": 1 }

3 9

Large RocksDB (db_slow_bytes) on OSD which is marked as out
by Wido den Hollander 31 Aug '20

31 Aug '20

Hello, On a Nautilus 14.2.8 cluster I am seeing large RocksDB database with many slow DB bytes in use. To investigate this further I marked one OSD as out and waited for the all the backfilling to complete. Once the backfilling was completed I exported BlueFS and investigated the RocksDB using 'ceph-kvstore-tool'. This resulted in 22GB of data. Listing all the keys in the RocksDB shows me there are 747.000 keys in the DB. A small portion are osdmaps, but the biggest amount are keys prefixed with 'b'. I dumped the stats of the RocksDB and this shows me: L1: 1/0: 439.32 KB L2: 1/0: 2.65 MB L3: 5/0: 14.36 MB L4: 127/0: 7.22 GB L5: 217/0: 13.73 GB Sum: 351/0: 20.98 GB So there is almost 21GB of data in this RocksDB database. Why? Where is this coming from? Throughout this cluster OSDs are suffering from many slow bytes used and I can't figure out why. Has anybody seen this or has a clue on what is going on? I have an external copy of this RocksDB database to do investigations on. Thank you, Wido

2 3

How to query status of scheduled commands.
by Frank Schilder 31 Aug '20

31 Aug '20

Hi all, can anyone help me with this? In mimic, for any of these commands: ceph osd [deep-]scrub ID ceph pg [deep-]scrub ID ceph pg repair ID an operation is scheduled asynchronously. How can I check the following states: 1) Operation is pending (scheduled, not started). 2) Operation is running. 3) Operation has completed. 4) Exit code and error messages if applicable. Many thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Re: osd regularly wrongly marked down
by Francois Legrand 31 Aug '20

31 Aug '20

We tried to rise the osd_memory_target from 4 to 8G but the problem still occurs (osd wrongly marked down few times a day). Does somebody have any clue ? F. > On Fri, Aug 28, 2020 at 10:34 AM Francois Legrand > <fleg(a)lpnhe.in2p3.fr <mailto:fleg@lpnhe.in2p3.fr>> wrote: > > Hi all, > > We have a ceph cluster in production with 6 osds servers (with > 16x8TB > disks), 3 mons/mgrs and 3 mdss. Both public and cluster > networks are in > 10GB and works well. > > After a major crash in april, we turned the option > bluefs_buffered_io to > false to workaround the large write bug when > bluefs_buffered_io was > true (we were in version 14.2.8 and the default value at this > time was > true). > Since that time, we regularly have some osds wrongly marked > down by the > cluster after heartbeat timeout (heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15). > > Generally the osd restart and the cluster is back healthy, but > several > time, after many of these kick-off the osd reach the > osd_op_thread_suicide_timeout and goes down definitely. > > We increased the osd_op_thread_timeout and > osd_op_thread_suicide_timeout... The problems still occurs > (but less > frequently). > > Few days ago, we upgraded to 14.2.11 and revert the timeout to > their > default value, hoping that it will solve the problem (we > thought that it > should be related to this bug > https://tracker.ceph.com/issues/45943) > but it didn't. We still have some osds wrongly marked down. > > Can somebody help us to fix this problem ? > Thanks. > > Here is an extract of an osd log at failure time: > > --------------------------------- > 2020-08-28 02:19:05.019 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 44.7d scrub starts > 2020-08-28 02:19:25.755 7f040e43d700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:19:25.755 7f040dc3c700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > this last line is repeated more than 1000 times > ... > 2020-08-28 02:20:17.484 7f040d43b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:20:17.551 7f03f1384700 0 > bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow > operation > observed for _collection_list, latency = 67.3532s, lat = 67s cid > =44.7d_head start GHMAX end GHMAX max 25 > ... > 2020-08-28 02:20:22.600 7f040dc3c700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:21:20.774 7f03f1384700 0 > bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow > operation > observed for _collection_list, latency = 63.223s, lat = 63s cid > =44.7d_head start > #44:beffc78d:::rbd_data.1e48e8ab988992.00000000000011bd:0# end > #MAX# max > 2147483647 > 2020-08-28 02:21:20.774 7f03f1384700 1 heartbeat_map > reset_timeout > 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 > 2020-08-28 02:21:20.805 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 44.7d scrub ok > 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) > log [WRN] : > Monitor daemon marked osd.16 down, but it is still running > 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) > log [DBG] : > map e609411 wrongly marked me down at e609410 > 2020-08-28 02:21:21.099 7f03fd997700 1 osd.16 609411 > start_waiting_for_healthy > 2020-08-28 02:21:21.119 7f03fd997700 1 osd.16 609411 start_boot > 2020-08-28 02:21:21.124 7f03f0b83700 1 osd.16 pg_epoch: 609410 > pg[36.3d0( v 609409'481293 (449368'478292,609409'481293] > local-lis/les=609403/609404 n=154651 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609410/609410/608752) > [25,72] r=-1 > lpr=609410 pi=[609403,609410)/1 luod=0'0 lua=609392'481198 > crt=609409'481293 lcod 609409'481292 active mbc={}] > start_peering_interval up [25,72,16] -> [25,72], acting > [25,72,16] -> > [25,72], acting_primary 25 -> 25, up_primary 25 -> 25, role 2 > -> -1, > features acting 4611087854031667199 upacting 4611087854031667199 > ... > 2020-08-28 02:21:21.166 7f03f0b83700 1 osd.16 pg_epoch: 609411 > pg[36.56( v 609409'480511 (449368'477424,609409'480511] > local-lis/les=609403/609404 n=153854 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609410/609410/609410) > [103,102] > r=-1 lpr=609410 pi=[609403,609410)/1 crt=609409'480511 lcod > 609409'480510 unknown NOTIFY mbc={}] state<Start>: > transitioning to Stray > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > public network em1 numa node 0 > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > cluster network em2 numa node 0 > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > objectstore and network numa nodes do not match > 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 > set_numa_affinity > not setting numa affinity > 2020-08-28 02:21:21.566 7f040a435700 1 osd.16 609413 tick > checking mon > for new map > 2020-08-28 02:21:22.515 7f03fd997700 1 osd.16 609414 state: > booting -> > active > 2020-08-28 02:21:22.515 7f03f0382700 1 osd.16 pg_epoch: 609414 > pg[36.20( v 609409'483167 (449368'480117,609409'483167] > local-lis/les=609403/609404 n=155171 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609414/609414/609361) > [97,16,72] > r=1 lpr=609414 pi=[609403,609414)/1 crt=609409'483167 lcod > 609409'483166 > unknown NOTIFY mbc={}] start_peering_interval up [97,72] -> > [97,16,72], > acting [97,72] -> [97,16,72], acting_primary 97 -> 97, > up_primary 97 -> > 97, role -1 -> 1, features acting 4611087854031667199 upacting > 4611087854031667199 > ... > 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 > pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] > local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) > [16,34,21] > r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod > 609409'479795 > mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up > [34,21] -> > [16,34,21], acting [34,21] -> [16,34,21], acting_primary 34 -> > 16, > up_primary 34 -> 16, role -1 -> 0, features acting > 4611087854031667199 > upacting 4611087854031667199 > 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 > pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] > local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c > 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) > [16,34,21] > r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod > 609409'479795 > mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary > 2020-08-28 02:21:24.738 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 36.2f3 scrub starts > 2020-08-28 02:22:18.857 7f03f1384700 0 log_channel(cluster) > log [DBG] : > 36.2f3 scrub ok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> >

1 0

How to repair rbd image corruption
by Jared 31 Aug '20

31 Aug '20

Hi, I have a rook cluster running with ceph 12.2.7 for almost one year. Recently some pvc couldn’t be attached with error as below, Warning FailedMount 7m19s kubelet, 192.168.34.119 MountVolume.SetUp failed for volume "pvc-8f4ca7ac-42ab-11ea-99d7-005056b84936" : mount command failed, status: Failure, reason: failed to mount volume /dev/rbd1 [ext4] to /var/lib/kubelet/plugins/rook.io/rook-ceph/mounts/pvc-8f4ca7ac-42ab-11ea-99d7-005056b84936, error 'fsck' found errors on device /dev/rbd1 but could not correct them: fsck from util-linux 2.23.2 /dev/rbd1: recovering journal /dev/rbd1 contains a file system with errors, check forced. /dev/rbd1: Inode 393244, end of extent exceeds allowed value (logical block 512, physical block 12091904, len 4388) After mapping storage to block device and run "fsck -y" on it, it indicated that filesystem was clean. Then retry to mount the storage, it still reported the same error as above. Response from "ceph status” is as below, cluster: id: 54a729b6-7b59-4e5b-bc09-7dc99109cbad health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set Degraded data redundancy: 50711/152133 objects degraded (33.333%), 100 pgs degraded, 100 pgs undersized mons rook-ceph-mon41,rook-ceph-mon44 are low on available space services: mon: 3 daemons, quorum rook-ceph-mon44,rook-ceph-mon47,rook-ceph-mon41 mgr: a(active) osd: 3 osds: 3 up, 3 in flags noscrub,nodeep-scrub data: pools: 1 pools, 100 pgs objects: 50711 objects, 190 GB usage: 383 GB used, 1111 GB / 1495 GB avail pgs: 50711/152133 objects degraded (33.333%) 100 active+undersized+degraded io: client: 78795 B/s wr, 0 op/s rd, 11 op/s wr From output, why is scrub disabled? Could I trigger it manually? And how could I check which pgs or objects is corrupted? Any advice for fixing the issue? Thanks for your help. Jared

1 0

Re: Erasure coding RBD pool for OpenStack
by Lazuardi Nasution 31 Aug '20

31 Aug '20

Hi Konstantin, I hope you or anybody still follows this old thread. Can this EC data pool be configured per pool, not per client? If we follow https://docs.ceph.com/docs/master/rbd/rbd-openstack/ we may see that cinder client will access vms and volumes pools, both with read and write permission. How can we handle this? If we config with different clients for nova (vms) and cinder (volumes), I think there will be a problem if there is cross pool access, especially on write. Let's say that client nova will create volume on instance creation for booting from that volume. Any thoughts? Best regards, > Date: Wed, 11 Jul 2018 11:16:27 +0700 > From: Konstantin Shalygin <k0ste(a)k0ste.ru> > To: ceph-users(a)lists.ceph.com > Subject: Re: [ceph-users] Erasure coding RBD pool for OpenStack > Glance, Nova and Cinder > Message-ID: <069ac368-22b0-3d18-937b-70ce39287cb1(a)k0ste.ru> > Content-Type: text/plain; charset=utf-8; format=flowed > > > So if you want, two more questions to you : > > > > - How do you handle your ceph.conf configuration (default data pool by > > user) / distribution ? Manually, config management, openstack-ansible... > > ? > > - Did you made comparisons, benchmarks between replicated pools and EC > > pools, on the same hardware / drives ? I read that small writes are not > > very performant with EC. > > > ceph.conf with default data pool is only need for Cinder at image > creation time, after this luminous+ rbd client will be found feature > "data-pool" and will perform data-io to this pool. > > > > # rbd info > > erasure_rbd_meta/volume-09ed44bf-7d16-453a-b712-a636a0d3d812 <----- > > meta pool ! > > rbd image 'volume-09ed44bf-7d16-453a-b712-a636a0d3d812': > > ??????? size 1500 GB in 384000 objects > > ??????? order 22 (4096 kB objects) > > ??????? data_pool: erasure_rbd_data??????? <----- our data pool > > ??????? block_name_prefix: rbd_data.6.a2720a1ec432bf > > ??????? format: 2 > > ??????? features: layering, exclusive-lock, object-map, fast-diff, > > deep-flatten, data-pool????????? <----- "data-pool" feature > > ??????? flags: > > ??????? create_timestamp: Sat Jan 27 20:24:04 2018 > > > > k > > > > ------------------------------ >

2 8

Re: Speeding up reconnection
by William Edwards 31 Aug '20

31 Aug '20

I replaced the VMs taking care of routing between clients and MDSes by physical machines. Problems below are solved. It seems to have been related to issues with the virtual NIC. It seemed to work well with E1000 instead of VirtIO... Met vriendelijke groeten, William Edwards ----- Original Message ----- From: William Edwards (wedwards(a)cyberfusion.nl) Date: 08/11/20 11:38 To: ceph-users(a)ceph.io Subject: Speeding up reconnection Hello, When connection is lost between kernel client, a few things happen: 1. Caps become stale: Aug 11 11:08:14 admin-cap kernel: [308405.227718] ceph: mds0 caps stale 2. MDS evicts client for being unresponsive: MDS log: 2020-08-11 11:12:08.923 7fd1f45ae700 0 log_channel(cluster) log [WRN] : evicting unresponsive client admin-cap.cf.ha.cyberfusion.cloud:DB0001-cap (144786749), after 300.978 seconds Client log: Aug 11 11:12:11 admin-cap kernel: [308643.051006] ceph: mds0 hung 3. Socket is closed: Aug 11 11:22:57 admin-cap kernel: [309289.192705] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state OPEN) I am not sure whether the kernel client or MDS closes the connection. I think the kernel client does so, because nothing is logged at the MDS side at 11:22:57 4. Connection is reset by MDS: MDS log: 2020-08-11 11:22:58.831 7fd1f9e49700 0 --1- [v2:[fdb7:b01e:7b8e:0:10:10:10:1]:6800/3619156441,v1:[fdb7:b01e:7b8e:0:10:10:10:1]:6849/3619156441] >> v1:[fc00:b6d:cfc:951::7]:0/133007863 conn(0x55bfaf1c2880 0x55c16cb47000 :6849 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept we reset (peer sent cseq 1), sending RESETSESSION Client log: Aug 11 11:22:58 admin-cap kernel: [309290.058222] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 connection reset 5. Kernel client reconnects: Aug 11 11:22:58 admin-cap kernel: [309290.058972] ceph: mds0 closed our session Aug 11 11:22:58 admin-cap kernel: [309290.058973] ceph: mds0 reconnect start Aug 11 11:22:58 admin-cap kernel: [309290.069979] ceph: mds0 reconnect denied Aug 11 11:22:58 admin-cap kernel: [309290.069996] ceph: dropping file locks for 000000006a23d9dd 1099625041446 Aug 11 11:22:58 admin-cap kernel: [309290.071135] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state NEGOTIATING) Question: As you can see, there's 10 minutes between losing the connection and the reconnection attempt (11:12:08 - 11:22:58). I could not find any settings related to the period after which reconnection is attempted. I would like to change this value from 10 minutes to something like 1 minute. I also tried searching the Ceph docs for the string '600' (10 minutes), but did not find anything useful. Hope someone can help. Environment details: Client kernel: 4.19.0-10-amd64 Ceph version: ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable) Met vriendelijke groeten, William Edwards

1 0

issues with object-map in benji
by Pavel Vondřička 31 Aug '20

31 Aug '20

Hello, Lately, we upgraded Ceph to version 15.2.4 and shortly after that we had a blackout, which caused a restart of all servers at once (BTW, Ceph did not come up well itself). Since then we were receiving lots of complaints about problems with "object-maps" with every benji backup (tool for differential backups of RBD images). So, I turned on "exclusive-lock" and "fast-diff" and rebuilt object-maps for all RBD images. Since then I am receiving the following message for every RBD image backup: 2020-08-24T06:25:10.451+0200 7f0b857fa700 -1 librbd::object_map::InvalidateRequest: 0x7f0b8000a690 invalidating object map in-memory 2020-08-24T06:25:10.451+0200 7f0b857fa700 -1 librbd::object_map::InvalidateRequest: 0x7f0b8000a690 invalidating object map on-disk 2020-08-24T06:25:10.451+0200 7f0b857fa700 -1 librbd::object_map::InvalidateRequest: 0x7f0b8000a690 should_complete: r=0 I am not sure whether it indicates some real problem limiting the functionality of the backups, or whether it is just natural part of the process. Is it a problem of Ceph and the unexpected blackout causing some corruption, or a problem of Benji and some new feature/issue of Ceph version 15? I wonder whether the object-maps were used before, since there were never any reports about any problems of this kind, and even the features mentioned above were (probably?) never turned on before (so there were no object maps at all?). I am quite confused. Thanks for any explanation, Pavel

1 0

Best Assignment Help Service In Australia
by jeniferbrooks027＠gmail.com 30 Aug '20

30 Aug '20

MyAssignmentHelpNow is famous for providing quality assignment help services at reasonable prices. We have been serving quality writing service on various subjects to reach out the maximum number of students in Australia by ensuring that our prices remain cheap and affordable. Our experts are quite skilled and knowledgeable to accomplish challenging tasks and we always strive to help you with best and cheap assignment writing service in Australia. https://myassignmenthelpnow.com/assignment-help-sydney/

2 1

Undo ceph osd destroy?
by Michael Fladischer 30 Aug '20

30 Aug '20

Hi, I accidentally destroyed the wrong OSD in my cluster. It is now marked as "destroyed" but the HDD is still there and data was not touch AFAICT. I was able to avtivate it again using ceph-volume lvm activate and I can make the OSD as "in" but it's status is not changing from "destroyed". Is there a way to unmark it so I can reintegrate it in the cluster? Regards, Michael

2 3

Why online thesis help is the best option?
by kayleebrown402＠gmail.com 30 Aug '20

30 Aug '20

When you choose the best thesis help to write your thesis, you can either hire a person or a professional writing agency like https://www.edumagnate.com/thesis-help.html. But whom to choose for your case! Well, both of them are equally good and advantageous. You can choose any one of them as per your requirement and as per your pocket permits. In the case of a writing agency, lots of writers are available. Some of those writers are high end, and some are affordable. You will get service as per the money paid by you. But in the case of the freelancer, you need to work with him only. But lots of freelancers are out there, and you can choose someone that fits your bill. Maximum thesis helpers work on several projects simultaneously. So, when you approach the person with your project, he will take that up and may not provide you the project before the deadline ends. There’s always a risk factor going around these people. You can always check online thesis help reviews on different platforms. Check what others are saying about the writer before hiring him for your project. But if you choose the writing agency, they will replace the writer or refund you if the project is not complete before the deadline.

1 0

Why can’t I get money off Cash app without card?
by david william 29 Aug '20

29 Aug '20

Cash app comes with the Visa debit card to make your transaction reliable as well as safe. Here we describe you to resolve your query get money off Cash app without card. Actually, if you lost your card or you didn’t get a Cash card yet, then you have to make a request for getting a Cash card.https://www.cashapp-customerservice.com/blog/get-money-cash-app-withou…

1 0

Re: Is it possible to mount a cephfs within a container?
by steven prothero 29 Aug '20

29 Aug '20

Hello, octopus 15.2.4 just as a test, I put my OSDs each inside of a LXD container. Set up cephFS and mounted it inside a LXD container and it works.

2 1

The most trusted Quickbooks technical support number
by david beckam 29 Aug '20

29 Aug '20

Quickbooks has changed the lives of thousands of its users by taking up the responsibility of bookkeeping for them. Keeping accounts becomes a breeze with business owners easily meeting tax compliance requirements. If you are new to Quickbooks, it is natural for you to face some difficulties during the learning curve. However, there is no need to worry as you can use our popular and trusted Quickbooks technical support number. Dial 1-877-916-7666, the toll free number of our tech support agency, and talk to our qualified professional. He will help you to overcome your Quickbooks related issues without any delay. https://www.assistanceforall.com/services/quickbook-support/

1 0

Saints redeemed secondary certain vs. superior-run Rams
by montgomeryqian＠yhwebmail.com 29 Aug '20

29 Aug '20

METAIRIE, La. Marshon Lattimore and the Saints secondary no for a longer time appear or strong haunted by way of the Minnesota Wonder touchdown go that stunningly finished their playoffs 1 12 months back.We use t anxiety. We wear t do none of that, Lattimore mentioned. We transform very well.Lattimore sponsored that up with his late-recreation interception towards Philadelphia ultimate weekend his instant of the activity securing the Saints holiday vacation in the direction of Sunday NFC name activity. At the moment the speculate is no matter whether a Saints defensive backfield that rated 29th in opposition to the go this time is penned towards do what it normally takes in direction of gradual down the strong Los Angeles Rams with the Tremendous Bowl bid upon the line.We re geared up, stated Saints cornerback Eli Apple, who was introduced within by means of exchange towards assistance shore up a Refreshing Orleans move safety that was shaky early this time.We ve accurately received in the direction of carry on toward be serious inside of such conferences with our thoughts, anyone choose upon the identical webpage and studying, simply working with it upon the industry and transferring difficult.Fresh new Orleans gave up an ordinary of 268.9 yards for every sport this year, still Lattimore reported element of their unflattering position as a go safety stems against a couple of of undesirable performances early inside of the period, starting up with a 48-40 decline towards Tampa Bay in just 7 days 1 and a 43-37 7 days 3 additional time victory at Atlanta <a href="https://www.ramsgearfanstore.com/Sam_Shields_Jersey-454">https://www.ramsgearfanstore.com/Sam_Shields_Jersey-454</a>.We experienced a substantial gap in the direction of climb out of at the starting off of the time. We use t incredibly pressure in excess of that whilst, Lattimore mentioned. We re successful. As a result, we re Terrific.Lattimore explained he additionally includes better separately because having difficulties some at the starting off of the time and may possibly be enjoying his excellent already.I m only manufacturing performs that occur my direction <a href="https://www.ramsgearfanstore.com/John_Kelly_Jersey-314">John Kelly Jersey</a>, Lattimore stated. I comprise in direction of be the one particular in the direction of create individuals performs. It a greater place immediately by now. Therefore, I comprise in the direction of appear up and do the activity for my workers.Saints nickel back again P.J. Williams explained the Saints terrible rating versus the go is deceptive mainly because Fresh Orleans rated initially utmost of the year towards the work just before completing moment. In the meantime, Drew Brees and Fresh new Orleans significant-driven offense include staked the Saints towards a good deal of prospects.And Although groups drop at the rear of, they are inclined in direction of operate significantly less since it requires much too considerably season off the clock.They can t work the ball. Your self acquired toward move the ball. On your own re participating in towards guiding, Williams claimed. Consequently a employees could possibly receive 300 or consequently passing yards, nevertheless at the finish of the working day, they re attempting towards arrive back again.Williams reported the Saints are further nervous with opposing QBs completion value than how plenty of yards they let <a href="https://www.ramsgearfanstore.com/Earnest_Edwards_Jersey-2286">https://www.ramsgearfanstore.com/Earnest_Edwards_Jersey-2286</a>. Upon that entrance, Contemporary Orleans rated rather superior, at 23rd, letting a 66.5 per cent completion value.Rams quarterback Jared Goff finished 28 of 40 passes for 391 yards and a few TDs the very last year he performed in just the Superdome.Irrespective of whether he ll be as effective inside the rematch relies upon upon how Los Angeles establishes toward assault the Saints, and how Clean Orleans responds.The Rams constantly want the working recreation, and rushed 48 days for 273 yards in just a playoff victory above Dallas previous 7 days <a href="https://www.ramsgearfanstore.com/Eric_Banks_Jersey-2331">https://www.ramsgearfanstore.com/Eric_Banks_Jersey-2331</a>.Yet Los Angeles mainly deserted the work throughout its midseason convention with the Saints immediately after Fresh new Orleans raced in direction of a 35-14 contribute within just the instant quarter. The Rams wound up dashing specifically 19 periods in just a 45-35 decline.Lattimore reported if the Rams deserted the work once again, Fresh Orleans secondary will comprise toward be written for large doses of receivers Brandin Chefs, whose pace results in him an elite deep hazard, and Robert Woods, who strains up all previously mentioned the business.And if the video game is conclusion, it could possibly be up toward the Saints secondary in the direction of conclusion out a further significant-stakes affair.Very last 12 months, that situation finished with the Vikings Stefon Diggs manufacturing a jumping capture close to the sideline and working absolutely free toward the stop zone as season expired when protection Marcus Williams disregarded a deal with and took out teammate Ken Crawley inside of the course of action <a href="https://www.ramsgearfanstore.com/Greg_Gaines_Jersey-1122">https://www.ramsgearfanstore.com/Greg_Gaines_Jersey-1122</a>.This calendar year, the Saints secondary consists of shut out a person playoff victory and seems guaranteed it will increase toward that situation once more, if necessary.We ve finished a best endeavor completing video games, creating modifications, taking part in quick, bodily, developing ideal performs upon the ball, exactly received toward continue to keep it transferring, Apple explained. One thing we normally communicate pertaining to is remaining online games. We require that upon our shoulders and that worry that arrives with it.

1 0

In opposition to large atop Kilimanjaro, Haloti Ngata retires
by Montgomeryqian＠yhwebmail.com 29 Aug '20

29 Aug '20

Haloti Ngata found out a one of a kind course towards announce his retirement towards the NFL versus atop Mount Kilimanjaro.The 35-12 months-aged Ngata, who performed with the Eagles past year immediately after a few yrs with the Lions, was at his perfect with Baltimore. He produced 2 All-Professional groups and 5 Qualified Bowls with the Ravens inside of 9 seasons as soon as getting a to start with-spherical draft decide on inside of 2006 [url=https://www.ravensgearfanstore.com/Broderick_Washington_Jr_Jersey-1586]… Washington Jr. Jersey[/url]. He gained a Tremendous Bowl with them within just February 2013.The defensive take care of printed his retirement concept Monday upon Instagram. Accurately a male track record upon greatest of the environment with a centre complete of gratitude, Ngata wrote. Thank on your own Lord for making it possible for me engage in the sport I appreciate for 13 memorable many years. I m retiring upon greatest. I may possibly be completed taking part in soccer, yet I m keeping restricted in direction of the friendships, recollections and knowledge I ve obtained together the course. No one who at any time gave his most straightforward regretted it. Going for walks absent with no regrets, specifically rest inside of being familiar with I gave it my all and experienced a helluva whole lot of enjoyable undertaking it.Stats have been not the evaluate for Ngata, who tied up blockers upon the inner and freed avid gamers these as Ray Lewis and Terrell Suggs in the direction of do their harm for the Ravens [url=https://www.ravensgearfanstore.com/Alvin_Jones_Jersey-979]https://www.r…] He way too includes been praised in the course of the a long time for his management abilities.

1 0

rados client connection to cluster timeout and debugging.
by Daniel Mezentsev 28 Aug '20

28 Aug '20

Hi all, I continue to work on my rados client, implementing error handling and trying to reconnect to cluster if it's not available. Here is my tests. Client is not connected to network, so cluster is not available, i configured client_mount_timeout = 0.5 (which is 5 seconds) and getting reasonable error - 110, Connection timed out. Then client is doing second and third attempt and in these cases error is a bit strange - 115, Operation now in progress. So failed rados_connect didn't restore state of connection - bug ? I was trying to enable debug: "log file" "./client.log" "log to file" "true" "debug ms" "10/10" "debug rados" "10/10" "debug objecter" "10/10" "debug monc" "10/10" "debug client" "10/10" But no luck to get anything in the log file. Running ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) under archlinux. Help is more then welcome. Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

1 1

Re: OSDs get full with bluestore logs
by Janne Johansson 28 Aug '20

28 Aug '20

Den fre 28 aug. 2020 kl 11:47 skrev Khodayar Doustar <khodayard(a)gmail.com>: > I've actually destroyed the cluster and a new one installed. > I've just changed the installation method and version. I've used > ceph-ansible this time and installed Nautilus. > The cluster worked fine with the same hardware. > Yes Janne, you are right that it had very small disks (9X20GB disks, 3 for > each node) but there was no problem with Nautilus. > > I just don't think anyone finds it useful to spend time figuring out the lowest possible limits for each release, for each type of OSD storage. So 10G worked at some point, 20 at another but if you are serious about ceph, make the disks thin provisioned and larger by a huge margin (say 100G) and let it grow to the size it needs for your tests. If the limit is 22 or 23.7 won't matter then and your test will run through. -- May the most significant bit of your life be positive.

1 0

How to change the pg numbers
by norman 28 Aug '20

28 Aug '20

Hi guys, I have a rbd pool pg_num 2048, I want to change it to 4096, how can I do this? If I change it directly to 4096, it may cause the client slow requests, What the better step size should be? Thanks, Kern

6 12

Re: Add OSD with primary on HDD, WAL and DB on SSD
by Zhenshi Zhou 28 Aug '20

28 Aug '20

In my deployment I part the disk for wal and db seperately as I can assign the size manually. For example, I assign every two partitions of nvme with 30G and 2G, for each osd's wal and db. Tony Liu <tonyliu0592(a)hotmail.com> 于2020年8月28日周五上午1:53写道： > How's WAL utilize disk when it shares the same device with DB? > Say device size 50G, 100G, 200G, they are no difference to DB > because DB will take 30G anyways. Does it make any difference > to WAL? > > Thanks! > Tony > > -----Original Message----- > > From: Zhenshi Zhou <deaderzzs(a)gmail.com> > > Sent: Wednesday, August 26, 2020 11:16 PM > > To: Tony Liu <tonyliu0592(a)hotmail.com> > > Cc: Anthony D'Atri <anthony.datri(a)gmail.com>; ceph-users(a)ceph.io > > Subject: Re: [ceph-users] Re: Add OSD with primary on HDD, WAL and DB on > > SSD > > > > Official document says that you should allocate 4% of the slow device > > space for block.db. > > > > But the main problem is that Bluestore uses RocksDB and RocksDB puts a > > file on the fast device only if it thinks that the whole layer will fit > > there. > > > > As for RocksDB, L1 is about 300M, L2 is about 3G, L3 is near 30G, and L4 > > is about 300G. > > For instance, RocksDB puts L2 files to block.db only if it’s at least 3G > > there. > > As a result, 30G is a acceptable value. > > > > Tony Liu <tonyliu0592(a)hotmail.com <mailto:tonyliu0592@hotmail.com> > > > 于2020年8月25日周二上午10:49写道： > > > > > > > -----Original Message----- > > > From: Anthony D'Atri <anthony.datri(a)gmail.com > > <mailto:anthony.datri@gmail.com> > > > > Sent: Monday, August 24, 2020 7:30 PM > > > To: Tony Liu <tonyliu0592(a)hotmail.com > > <mailto:tonyliu0592@hotmail.com> > > > > Subject: Re: [ceph-users] Re: Add OSD with primary on HDD, WAL > > and DB on > > > SSD > > > > > > Why such small HDDs? Kinda not worth the drive bays and power, > > instead > > > of the complexity of putting WAL+DB on a shared SSD, might you > > have been > > > able to just buy SSDs and not split? ymmv. > > > > 2TB is for testing, it will bump up to 10TB for production. > > > > > The limit is a function of the way the DB levels work, it’s not > > > intentional. > > > > > > WAL by default takes a fixed size, like 512 MB or something. > > > > > > 64 GB is a reasonable size, it accomodates the WAL and allows > > space for > > > DB compaction without overflowing. > > > > For each 10TB HDD, what's the recommended DB device size for both > > DB and WAL? The doc recommends 1% - 4%, meaning 100GB - 400GB for > > each 10TB HDD. But given the WAL data size and DB data size, I am > > not sure if that 100GB - 400GB will be used efficiently. > > > > > With this commit the situation should be improved, though you > > don’t > > > mention what release you’re running > > > > > > https://github.com/ceph/ceph/pull/29687 > > > > I am using ceph version 15.2.4 octopus (stable). > > > > Thanks! > > Tony > > > > > >>> I don't need to create > > > >>> WAL device, just primary on HDD and DB on SSD, and WAL will > > be using > > > >>> DB device cause it's faster. Is that correct? > > > >> > > > >> Yes. > > > >> > > > >> > > > >> But be aware that the DB sizes are limited to 3GB, 30GB and > > 300GB. > > > >> Anything less than those sizes will have a lot of untilised > > space, > > > >> e.g a 20GB device will only utilise 3GB. > > > > > > > > I have 1 480GB SSD and 7 2TB HDDs. 7 LVs are created on SSD, > > each is > > > > about 64GB, for 7 OSDs. > > > > > > > > Since it's shared by DB and WAL, DB will take 30GB and WAL will > > take > > > > the rest 34GB. Is that correct? > > > > > > > > Is that size of DB and WAL good for 2TB HDD (block store and > > object > > > > store cases)? > > > > > > > > Could you share a bit more about the intention of such limit? > > > > > > > > > > > > Thanks! > > > > Tony > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph- > > users(a)ceph.io> To unsubscribe send an > > > > email to ceph-users-leave(a)ceph.io <mailto:ceph-users- > > leave(a)ceph.io> > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph- > > users(a)ceph.io> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > <mailto:ceph-users-leave@ceph.io> > > > >

2 1

Fwd: Ceph Upgrade Issue - Luminous to Nautilus (14.2.11 ) using ceph-ansible
by Suresh Rama 28 Aug '20

28 Aug '20

Hi All, We encountered an issue while upgrading our Ceph cluster from Luminous 12.2.12 to Nautilus 14.2.11. We used https://docs.ceph.com/docs/master/releases/nautilus/#upgrading-from-mimic-o… and ceph-ansible to upgrade the cluster. We use HDD for data and NVME for WAL and DB. *Cluster Background:* HP DL360 24 x 3.6T SATA 2x1.6T NVME for Journal osd_scenario: non-collocated current version: Luminous 12.2.12 & 12.2.5 type: bluestore The upgrade went well for MONs (though I had to overcome the systemd masking issues). While testing OSD upgrade with one OSD node, we encountered issue with OSD daemon failing quickly after startup. After comparing and checking the block devices mapping, everything looks fine. The nodes was up for more than 700+ days and then I decided to do a clean reboot. After that noticed the mount points are completely missing and also ceph-disk is no longer part of nautilus. Had to manually mount the partitions after checking disk partitions and whoami information. After manually mounting the osd.108, now it's throwing permission error which I'm still reviewing (bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Permission denied). Enclosed the log of the OSD for full review - https://pastebin.com/7k0xBfDV. *Questions*: What could have went wrong here and how can we fix this? Do we need to migrate the Luminous cluster from ceph-disk to ceph-volume before attempting upgrade or any other best practice can be followed? What's the best upgrade method using ceph-ansible to move from Luminous to Nautilus? Manual upgrade of Ceph-ansible? Started thinking now Octopus release which uses container, what is the best transition path for long run? We don't want to destroy and rebuild the entire cluster but we can do one node at a time but that would be a very lengthy process for 2500+ systems of 13 clusters. Looking for help and expert comments on the transition path. Any help would be greatly appreciated. 2020-08-27 14:41:01.132 7f0e0ebf2c00 0 bdev(0xb7e2a80 /var/lib/ceph/osd/ceph-108/block.wal) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-108/block.wal failed: (22) Invalid argument 2020-08-27 14:41:01.132 7f0e0ebf2c00 1 bdev(0xb7e2a80 /var/lib/ceph/osd/ceph-108/block.wal) open size 1073741824 (0x40000000, 1 GiB) block_size 4096 (4 KiB) non-rotational discard supported 2020-08-27 14:41:01.132 7f0e0ebf2c00 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-108/block.wal size 1 GiB 2020-08-27 14:41:01.132 7f0e0ebf2c00 0 set rocksdb option compaction_style = kCompactionStyleLevel 2020-08-27 14:41:01.132 7f0e0ebf2c00 -1 rocksdb: Invalid argument: Can't parse option compaction_threads 2020-08-27 14:41:01.136 7f0e0ebf2c00 -1 /build/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_upgrade_super()' thread 7f0e0ebf2c00 time 2020-08-27 14:41:01.135973 /build/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 10249: FAILED ceph_assert(ondisk_format > 0) ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x846368] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x846543] 3: (BlueStore::_upgrade_super()+0x4b6) [0xd62346] 4: (BlueStore::_mount(bool, bool)+0x592) [0xdb0b52] 5: (OSD::init()+0x3f3) [0x8f5483] 6: (main()+0x27e2) [0x84c462] 7: (__libc_start_main()+0xf0) [0x7f0e0bda3830] 8: (_start()+0x29) [0x880389] Journalctl -xu log ================ Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309 7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Perm Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Perm Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block) _read_bdev_label failed to op Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open got: (13) Perm Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 osd.108 0 OSD:init: unable to mount object store Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317 7fe9410bfc00 -1 ** ERROR: osd init failed: (13) Permission denied Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Main process exited, code=exited, status=1/FAILURE Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Unit entered failed state. Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Failed with result 'exit-code'. Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Service hold-off time over, scheduling restart. Aug 27 20:18:39 pistore-as-b03 systemd[1]: Stopped Ceph object storage daemon osd.108. Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Start request repeated too quickly. Aug 27 20:18:39 pistore-as-b03 systemd[1]: Failed to start Ceph object storage daemon osd.108. Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Aug 27 20:18:40 pistore-as-b03 systemd[1]: [/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' -- Regards, Suresh Note: Resent as the size was big and waiting on moderator approval

2 1

issue with monitors
by techno10＠verizon.net 28 Aug '20

28 Aug '20

i'm running the following [root@node1 ~]# ceph --versionceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) on fedora 32.. installed from the built-in repos. I'm running into a simple issue that's rather frustrating. Here is a set of commands i'm running and output: [root@node1 ~]# ceph orch daemon add mon node3:[v2:172.16.0.47:3000,v1:172.16.0.47:6789]Deployed mon.node3 on host 'node3' [root@node1 ~]# ceph mon dumpdumped monmap epoch 4epoch 4fsid c43406b4-e8c2-11ea-b934-001b21d6d88clast_changed 2020-08-28T00:13:27.040659+0000created 2020-08-28T00:10:04.265004+0000min_mon_release 15 (octopus)0: [v2:172.16.0.45:3300/0,v1:172.16.0.45:6789/0] mon.node11: [v2:172.16.0.46:3300/0,v1:172.16.0.46:6789/0] mon.node2 it said it added it but it fails to show up? I attempt to re-add: [root@node1 ~]# ceph orch daemon add mon node3:[v2:172.16.0.47:3000,v1:172.16.0.47:6789]Error ENOENT: ('name %s already in use', 'node3') but again, it's not there, checking with a different command: [root@node1 ~]# ceph mon state4: 2 mons at {node1=[v2:172.16.0.45:3300/0,v1:172.16.0.45:6789/0],node2=[v2:172.16.0.46:3300/0,v1:172.16.0.46:6789/0]}, election epoch 16, leader 0 node1, quorum 0,1 node1,node2 this is a very basic function that I think I'm doing correctly but it seems like it's not working properly?? thanks,Mike

1 0

Re: [cephadm] Deploy Ceph in a closed environment
by Tony Liu 27 Aug '20

27 Aug '20

Please discard this question, I figure it out. Tony > -----Original Message----- > From: Tony Liu <tonyliu0592(a)hotmail.com> > Sent: Thursday, August 27, 2020 1:55 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] [cephadm] Deploy Ceph in a closed environment > > Hi, > > I'd like to deploy Ceph in a closed environment (no connectivity to > public). I will build repository and registry to hold required packages > and container images. How do I specify the private registry when running > "cephadm bootstrap"? The same question for adding OSD. > > Thanks! > Tony > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

[cephadm] Deploy Ceph in a closed environment
by Tony Liu 27 Aug '20

27 Aug '20

Hi, I'd like to deploy Ceph in a closed environment (no connectivity to public). I will build repository and registry to hold required packages and container images. How do I specify the private registry when running "cephadm bootstrap"? The same question for adding OSD. Thanks! Tony

1 0

ceph auth ls
by Marc Roos 27 Aug '20

27 Aug '20

Am I the only one that thinks it is not necessary to dump these keys with every command (ls and get)? Either remove these keys from auth ls and auth get. Or remove the commands "auth print_key" "auth print-key" and "auth get-key"

1 1

Is it possible to mount a cephfs within a container?
by Marc Roos 27 Aug '20

27 Aug '20

I am getting this, on a osd node I am able to mount the path. adding ceph secret key to kernel failed: Operation not permitted

1 0

Fwd: Upgrade Path Advice Nautilus (CentOS 7) -> Octopus (new OS)
by Cloud Guy 27 Aug '20

27 Aug '20

Hello, Looking for a bit of guidance / approach to upgrading from Nautilus to Octopus considering CentOS and Ceph-Ansible. We're presently running a Nautilus cluster (all nodes / daemons 14.2.11 as of this post). - There are 4 monitor-hosts with mon, mgr, and dashboard functions consolidated; - 4 RGW hosts - 4 ODS costs, with 10 OSDs each. This is planned to scale to 7 nodes with additional OSDs and capacity (considering to do this as part of upgrade process) - Currently using ceph-ansible (however it's a process to maintain scripts / configs between playbook versions - although a great framework, not ideal in our case; - All hosts run CentOS 7.x; - dm-crypt in use on LVM OSDs (via ceph-ansible); - Deployment IS NOT containerized. Octopus support on CentOS 7 is limited due to python dependencies, as a result we want to move to CentOS 8 or Ubuntu 20.04. The other outlier is CentOS native Kernel support for LSI2008 (eg. 9211) HBAs which some of our OSD nodes use. Irrespective of OS considerations above, the upgrade will be to an OS that fully supports Octopus. We'd like to make use of ceph orchestrator for on-going cluster management. Here's an upgrade path scenario that is being considered. At a high-level: 1. Deploy a new monitor on CentOS 8. May be Nautilus via established ceph-ansible playbook. 2. Upgrade new monitor to Octopus using dnf / ceph package upgrade. 3. Decommission individual monitor hosts (existing on CentOS 7) and redeploy on CentOS 8 via ceph orchestrator from new monitor node; 4. Repeat until all monitors are on new OS + Octopus (all deployed via Ceph Orchestrator. 5. Add additional OSD nodes / drives / capacity via orchestrator on Octopus; 6. Upgrade existing OSD hosts by keeping OSDs intact, reinstalling new OS (CentOS 8 or Ubuntu 20.04); 7. Deploy ceph octopus on new nodes via orchestrator; 8. Reactivate / rescan in-tact OSDs on newly redeployed node. (i.e. ceph-volume lvm activate --all) 9. Rinse / repeat for remaining Nautilus nodes. 10. Manually upgrade RGW packages on gateway nodes. Thank you.

2 2

Re: Add OSD with primary on HDD, WAL and DB on SSD
by Zhenshi Zhou 27 Aug '20

27 Aug '20

Official document says that you should allocate 4% of the slow device space for block.db. But the main problem is that Bluestore uses RocksDB and RocksDB puts a file on the fast device only if it thinks that the whole layer will fit there. As for RocksDB, L1 is about 300M, L2 is about 3G, L3 is near 30G, and L4 is about 300G. For instance, RocksDB puts L2 files to block.db only if it’s at least 3G there. As a result, 30G is a acceptable value. Tony Liu <tonyliu0592(a)hotmail.com> 于2020年8月25日周二上午10:49写道： > > -----Original Message----- > > From: Anthony D'Atri <anthony.datri(a)gmail.com> > > Sent: Monday, August 24, 2020 7:30 PM > > To: Tony Liu <tonyliu0592(a)hotmail.com> > > Subject: Re: [ceph-users] Re: Add OSD with primary on HDD, WAL and DB on > > SSD > > > > Why such small HDDs? Kinda not worth the drive bays and power, instead > > of the complexity of putting WAL+DB on a shared SSD, might you have been > > able to just buy SSDs and not split? ymmv. > > 2TB is for testing, it will bump up to 10TB for production. > > > The limit is a function of the way the DB levels work, it’s not > > intentional. > > > > WAL by default takes a fixed size, like 512 MB or something. > > > > 64 GB is a reasonable size, it accomodates the WAL and allows space for > > DB compaction without overflowing. > > For each 10TB HDD, what's the recommended DB device size for both > DB and WAL? The doc recommends 1% - 4%, meaning 100GB - 400GB for > each 10TB HDD. But given the WAL data size and DB data size, I am > not sure if that 100GB - 400GB will be used efficiently. > > > With this commit the situation should be improved, though you don’t > > mention what release you’re running > > > > https://github.com/ceph/ceph/pull/29687 > > I am using ceph version 15.2.4 octopus (stable). > > Thanks! > Tony > > > >>> I don't need to create > > >>> WAL device, just primary on HDD and DB on SSD, and WAL will be using > > >>> DB device cause it's faster. Is that correct? > > >> > > >> Yes. > > >> > > >> > > >> But be aware that the DB sizes are limited to 3GB, 30GB and 300GB. > > >> Anything less than those sizes will have a lot of untilised space, > > >> e.g a 20GB device will only utilise 3GB. > > > > > > I have 1 480GB SSD and 7 2TB HDDs. 7 LVs are created on SSD, each is > > > about 64GB, for 7 OSDs. > > > > > > Since it's shared by DB and WAL, DB will take 30GB and WAL will take > > > the rest 34GB. Is that correct? > > > > > > Is that size of DB and WAL good for 2TB HDD (block store and object > > > store cases)? > > > > > > Could you share a bit more about the intention of such limit? > > > > > > > > > Thanks! > > > Tony > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Ceph Tech Talk: Secure Token Service in the Rados Gateway
by Mike Perez 27 Aug '20

27 Aug '20

Hi everyone, Join us August 27th at 17:00 UTC to hear Pritha Srivastava present on this month's Ceph Tech Talk: Secure Token Service in the Rados Gateway. Calendar invite and archive can be found here: https://ceph.io/ceph-tech-talks/ If you're interested or know someone who can present September 24th, or October 22nd please let me know! -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 1

[Ceph Octopus 15.2.3 ] MDS crashed suddenly
by carlimeunier＠gmail.com 27 Aug '20

27 Aug '20

Hi, I made a fresh install of Ceph Octopus 15.2.3 recently. And after a few days, the 2 standby MDS suddenly crashed with segmentation fault error. I try to restart it but it does not start. Here is the error : -20> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _renew_subs -19> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _send_mon_message to mon.2 at v1:172.31.36.98:6789/0 -18> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcf9530c0 version 269 -17> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa87520 version 269 -16> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa875c0 version 269 -15> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa871c0 version 269 -14> 2020-07-17T13:50:27.888+0000 7fc8c8c55700 10 monclient: get_auth_request con 0x559dcfada000 auth_method 0 -13> 2020-07-17T13:50:27.888+0000 7fc8c9456700 10 monclient: get_auth_request con 0x559dcfada800 auth_method 0 -12> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) recover start -11> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) read_head -10> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 4 mds.0.log Waiting for journal 0x200 to recover... -9> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_read_head loghead(trim 4194304, expire 4231216, write 4329405, stream_format 1). probing for end of log (from 4329405)... -8> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) probing for end of the log -7> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_probe_end write_pos = 4329949 (header had 4329405). recovered. -6> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Journal 0x200 recovered. -5> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Recovered journal 0x200 in format 1 -4> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 2 mds.0.0 Booting: 1: loading/discovering base inodes -3> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x100 -2> 2020-07-17T13:50:27.894+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x1 -1> 2020-07-17T13:50:27.894+0000 7fc8c0444700 2 mds.0.0 Booting: 2: replaying mds log 0> 2020-07-17T13:50:27.896+0000 7fc8bec41700 -1 *** Caught signal (Segmentation fault) ** in thread 7fc8bec41700 thread_name:md_log_replay Here is the cluster information : # ceph status cluster: id: dd024fe1-4996-4fed-ba57-03090e53724d health: HEALTH_WARN 20 daemons have recently crashed services: mon: 3 daemons, quorum 2,0,1 (age 2d) mgr: mgr.0(active, since 9d), standbys: mgr.2, mgr.1 mds: cephfs:1 {0=node0=up:active} 1 up:standby-replay 1 up:standby osd: 3 osds: 3 up (since 28h), 3 in (since 9d) task status: scrub status: mds.node0: idle mds.node2: idle data: pools: 3 pools, 49 pgs objects: 29 objects, 170 KiB usage: 3.0 GiB used, 41 TiB / 41 TiB avail pgs: 49 active+clean io: client: 853 B/s rd, 1 op/s rd, 0 op/s wr There is only 1 client connected to the cluster. Please, does anyone have any idea? Thanks

3 4

rados df with nautilus / bluestore
by Manuel Lausch 27 Aug '20

27 Aug '20

Hi we found a very ugly issue in rados df I have several clusters, all running ceph nautilus (14.2.11), We have there replicated pools with replica size 4. On the older clusters "rados df" shows in the used column the net used space. On our new cluster, rados df shows in the used column the gross used space. The older clusters was upgraded from luminous (and before) and uses filestore and the new cluster is initally deployed with nautilus and bluestore. why are the outputs different? Is this related with nautilus or with bluestore? For our reporting this values have a significant relevance and now I am running in such discrepancies. What commands/metrics can I use to get more reliable values. Maybe "ceph df detail"? Manuel

2 1

radowsgw still needs dedicated clientid?
by Marc Roos 27 Aug '20

27 Aug '20

I think I can remember reading somewhere that every radosgw is required to run with their own clientid. Is this still necessary? Or can I run multiple instances of radosgw with the same clientid? So can have something like rgw: 2 daemons active (rgw1, rgw1, rgw1)

2 2

Recover pgs from failed osds
by Vahideh Alinouri 27 Aug '20

27 Aug '20

vahideh.alinouri(a)gmail.com

1 0

kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay)
by Jake Grimmett 27 Aug '20

27 Aug '20

Dear all, After enabling "allow_standby_replay" on our cluster we are getting (lots) of identical errors on the client /var/log/messages like Apr 29 14:21:26 hal kernel: ceph: mdsmap_decode got incorrect state(up:standby-replay) We are using the ml kernel 5.6.4-1.el7 on Scientific Linux 7.8 Cluster and client are running Ceph v14.2.9 Setting was enabled with: # ceph fs set cephfs allow_standby_replay true [root@ceph-s1 ~]# ceph mds stat cephfs:1 {0=ceph-s3=up:active} 1 up:standby-replay 2 up:standby Is this something to worry about, or should we just disable allow_standby_replay ? any advice appreciated, many thanks Jake Note: I am working from home until further notice. For help, contact unixadmin(a)mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

2 2

export administration regulations issue for ceph community edition
by Peter Parker 27 Aug '20

27 Aug '20

Does anyone know of any new statements from the ceph community or foundation regarding EAR? I read the legal page of ceph.com and mentioned some information. https://ceph.com/legal-page/terms-of-service/ But I am still not sure, if my clients and I are within the scope of the entity list, whether the use of ceph community edition complies with the corresponding laws. Whether it affects the current release and future releases.

1 0

How To Configure Bellsouth Email Settings in a Right Way?
by sofi Hayat 27 Aug '20

27 Aug '20

It is required to use the right server and port settings to enjoy all the benefits of the Bellsouth email service. It is also recommended everyone to configure Bellsouth Email Settings and correctly and appropriately. There are few users unable to setup Bellsouth email on Android phone, iPhone, or computer device. For such helpless candidates, we provide helpline number by which they connect with top-most technicians for quality assistance. Once you contact to tech-savvy, your Bellsouth email settings will easily be configured in a second. https://www.emailsupport.us/blog/bellsouth-email-settings-for-outlook/

1 0

How To Configure Bellsouth Email Settings in a Right Way?
by sofi Hayat 27 Aug '20

27 Aug '20

It is required to use the right server and port settings to enjoy all the benefits of the Bellsouth email service. It is also recommended everyone to configure Bellsouth Email Settings and correctly and appropriately. There are few users unable to setup Bellsouth email on Android phone, iPhone, or computer device. For such helpless candidates, we provide helpline number by which they connect with top-most technicians for quality assistance. Once you contact to tech-savvy, your Bellsouth email settings will easily be configured in a second. https://www.emailsupport.us/blog/bellsouth-email-settings-for-outlook/

1 0

pg stuck in unknown state
by Michael Thomas 27 Aug '20

27 Aug '20

On my relatively new Octopus cluster, I have one PG that has been perpetually stuck in the 'unknown' state. It appears to belong to the device_health_metrics pool, which was created automatically by the mgr daemon(?). The OSDs that the PG maps to are all online and serving other PGs. But when I list the PGs that belong to the OSDs from 'ceph pg map', the offending PG is not listed. # ceph pg dump pgs | grep ^1.0 dumped pgs 1.0 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-08T09:30:33.251653-0500 0'0 0:0 [] -1 [] -1 0'0 2020-08-08T09:30:33.251653-0500 0'0 2020-08-08T09:30:33.251653-0500 0 # ceph osd pool stats device_health_metrics pool device_health_metrics id 1 nothing is going on # ceph pg map 1.0 osdmap e7199 pg 1.0 (1.0) -> up [41,40,2] acting [41,0] What can be done to fix the PG? I tried doing a 'ceph pg repair 1.0', but that didn't seem to do anything. Is it safe to try to update the crush_rule for this pool so that the PG gets mapped to a fresh set of OSDs? --Mike

4 5

iSCSI gateways in nautilus dashboard in state down
by Willi Schiegel 26 Aug '20

26 Aug '20

Hello All, I have a Nautilus (14.2.11) cluster which is running fine on CentOS 7 servers. 4 OSD nodes, 3 MON/MGR hosts. Now I wanted to enable iSCSI gateway functionality to be used by some Solaris and FreeBSD clients. I followed the instructions under https://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli-manual-install and https://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/ I setup two iSCSI targets and they both show up with gwcli on both gateways: [root@osd1 ~]# gwcli ls ... o- gateways ..................... [Up: 2/2, Portals: 2] | o- osd1.mydomain.pri ......... [172.29.1.171 (UP)] | o- osd2.mydomain.pri ......... [172.29.1.172 (UP)] ... tcmu-runner, rbd-target-gw and rbd-target-api are active (running) on both gateways, there is no firewall and SELinux is disabled but on the dashboard the state of the gateways is "down". and ceph-mgr.mon2.log shows mgr[dashboard] iscsi REST API failed GET req status: 403 Any hints? Thank you. Best Willi

3 3

Re: anyone using ceph csi
by Jason Dillaman 26 Aug '20

26 Aug '20

On Wed, Aug 26, 2020 at 10:33 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > >> > >> > >> I was wondering if anyone is using ceph csi plugins[1]? I would like > to > >> know how to configure credentials, that is not really described for > >> testing on the console. > >> > >> I am running > >> ./csiceph --endpoint unix:///tmp/mesos-csi-XSJWlY/endpoint.sock > --type > >> rbd --drivername rbd.csi.ceph.com --nodeid test > >> > >> Connection is fine > >> [ ~]# csc identity plugin-info > >> "rbd.csi.ceph.com" "canary" > >> > >> However I have no idea how to configure the clientid, pool etc in > the > >> volumes > >> > >> > >> [1] > >> https://github.com/ceph/ceph-csi > >> > >> Ps. I am not using kubernetes. > > > >The credentials and Ceph cluster configuration metadata are passed via > >the RPC calls as per the CSI spec. In k8s, these details would be > >stored in StorageClass and Secret objects. > > So there is no way of testing this driver via the commandline, with > some generic grpc client? > You would need some way to inject the correct/expected gRPC calls as per the CSI spec [1]. Data from the StorageClass mostly gets translated and sent via the "parameters" argument and data from the Secret gets translated and sent via the "secrets" argument of various gRPC requests. [1] https://github.com/container-storage-interface/spec/blob/master/spec.md -- Jason

1 0

Re: anyone using ceph csi
by Jason Dillaman 26 Aug '20

26 Aug '20

On Wed, Aug 26, 2020 at 10:11 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > I was wondering if anyone is using ceph csi plugins[1]? I would like to > know how to configure credentials, that is not really described for > testing on the console. > > I am running > ./csiceph --endpoint unix:///tmp/mesos-csi-XSJWlY/endpoint.sock --type > rbd --drivername rbd.csi.ceph.com --nodeid test > > Connection is fine > [ ~]# csc identity plugin-info > "rbd.csi.ceph.com" "canary" > > However I have no idea how to configure the clientid, pool etc in the > volumes > > > [1] > https://github.com/ceph/ceph-csi > > Ps. I am not using kubernetes. The credentials and Ceph cluster configuration metadata are passed via the RPC calls as per the CSI spec. In k8s, these details would be stored in StorageClass and Secret objects. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Jason

2 1

anyone using ceph csi
by Marc Roos 26 Aug '20

26 Aug '20

I was wondering if anyone is using ceph csi plugins[1]? I would like to know how to configure credentials, that is not really described for testing on the console. I am running ./csiceph --endpoint unix:///tmp/mesos-csi-XSJWlY/endpoint.sock --type rbd --drivername rbd.csi.ceph.com --nodeid test Connection is fine [ ~]# csc identity plugin-info "rbd.csi.ceph.com" "canary" However I have no idea how to configure the clientid, pool etc in the volumes [1] https://github.com/ceph/ceph-csi Ps. I am not using kubernetes.

1 0

cephfs needs access from two networks
by Simon Sutter 26 Aug '20

26 Aug '20

Hello, So I know, the mon services can only bind to just one ip. But I have to make it accessible to two networks because internal and external servers have to mount the cephfs. The internal ip is 10.99.10.1 and the external is some public-ip. I tried nat'ing it with this: "firewall-cmd --zone=public --add-forward-port=port=6789:proto=tcp:toport=6789:toaddr=10.99.10.1 -permanent" So the nat is working, because I get a "ceph v027" (alongside with some gibberish) when I do a telnet "telnet *public-ip* 6789" But when I try to mount it, I get just a timeout: mount -vvvv -t ceph *public-ip*:6789:/testing /mnt -o name=test,secretfile=/root/ceph.client. test.key mount error 110 = Connection timed out The tcpdump also recognizes a "Ceph Connect" packet, coming from the mon. How can I get around this problem? Is there something I have missed? Specs: Latest Octopus 15.2.4 Centos 8 8 Nodes No health warnings. Thanks in advance, Simon

2 1

can not remove orch service
by Ml Ml 26 Aug '20

26 Aug '20

Hello, root@ceph02:~# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID mgr.ceph01 ceph01 running (18m) 6s ago 4w 15.2.4 docker.io/ceph/ceph:v15.2.4 54fa7e66fb03 7deebe09f6fd (...) mgr.cph02 ceph02 error 3s ago 4w <unknown> docker.io/ceph/ceph:v15.2.4 <unknown> <unknown> I must have been drunk when i added "mgr.cph02". Now i am sober again, but i get: root@ceph02:~# ceph orch rm mgr.cph02 --force Failed to remove service. <mgr.cph02> was not found. :-( Any hint? Cheers, Michael

1 0

transit upgrade qithout mgr
by Dzianis Kahanovich 26 Aug '20

26 Aug '20

I have production cluster under jewel with rbd & mds under Gentoo. Building luminous with mgr now problematic (mostly by dropping python 3.5 on the eclass level). But for nautilus/etc I must go over luminous as transit. Can I temporary use luminous without mgr (at least to wait for scrub)? What may be happened? -- WBR, Dzianis Kahanovich AKA Denis Kaganovich, http://mahatma.bspu.by/

1 0

ceph-mon hanging when setting hdd osd's out
by maximilian.stinsky＠gec.io 25 Aug '20

25 Aug '20

Hi all, we are in the process of upgrading one of our three containerised ceph clusters from luminous(12.2.12) to nautilus(14.2.9). We already upgraded all ceph-mons, ceph-mgrs and all of our ceph-osd hosts that are full flash nodes with optane cache and nvme data devices. Our process to upgrade osds is to completely purge a complete host and redeploy the osd's with nautilus and lvm under it. This all went fine until we started touching our hdd nodes which serve a pool that provides a cephfs with erasure coding. The problem we are facing at the moment is that when we set a single hdd osd out the ceph command starts hanging for a couple of tens of seconds and the ceph quorum gets degraded because one of the ceph-mons gets marked as out with a lease_timeout. In the logs we could see some slow ops from the failing ceph-mon which were mon_subscribe events from osd's (full flash and hdd) hanging between all_read and dispatched for around 10 seconds. In our metrics we can see that the memory consumption of that one ceph-mon (not the leader!) increases to up to 60GB and also the cpu usage increases dramatically. Looking into logs does not show any obvious problem, we can see that the cluster sets the osd out and starts backfilling to other osds, but at some point the failing mon stops logging completely for around one minute then resumes logging after rejoining the quorum and keeps on logging normal backfilling behaviour. Here are some heap stats after the cluster was responsive and the ceph-mon was behaving normal again: MALLOC: 711286936 ( 678.3 MiB) Bytes in use by application MALLOC: + 27156045824 (25898.0 MiB) Bytes in page heap freelist MALLOC: + 17420216 ( 16.6 MiB) Bytes in central cache freelist MALLOC: + 9370880 ( 8.9 MiB) Bytes in transfer cache freelist MALLOC: + 25277104 ( 24.1 MiB) Bytes in thread cache freelists MALLOC: + 104857600 ( 100.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 28024258560 (26726.0 MiB) Actual memory used (physical + swap) MALLOC: + 75189862400 (71706.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 103214120960 (98432.7 MiB) Virtual address space used MALLOC: MALLOC: 33647 Spans in use MALLOC: 23 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. We tested all this in a test cluster without any problems. Does anyone has an idea what could be going on or where to look for further debugging? Best Regards Max

1 0

rgw-orphan-list
by Andrei Mikhailovsky 25 Aug '20

25 Aug '20

While continuing my saga with the rgw orphans and dozens of terabytes of wasted space I have used the rgw-orphan-list tool. after about 45 mins the tool has crashed ((( # time rgw-orphan-list .rgw.buckets Pool is ".rgw.buckets". Note: output files produced will be tagged with the current timestamp -- 202008241403. running 'rados ls' at Mon Aug 24 15:03:29 BST 2020 running 'radosgw-admin bucket radoslist' at Mon Aug 24 15:26:37 BST 2020 /usr/bin/rgw-orphan-list: line 64: 31745 Aborted (core dumped) radosgw-admin bucket radoslist > "$rgwadmin_out" 2> "$rgwadmin_err" An error was encountered while running 'radosgw-admin radoslist'. Aborting. Review file './radosgw-admin-202008241403.error' for details. *** *** WARNING: The results are incomplete. Do not use! *** *** I've got the error file with more information on the error if anyone is interested in improving the tool. Cheers Andrei

1 1

Re: RBD volume QoS support
by Jason Dillaman 25 Aug '20

25 Aug '20

On Tue, Aug 25, 2020 at 6:54 AM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Ceph folks, > > I am running Openstack Queens to host a variety of Apps, with ceph backend storage Luminous 12.2.13. > > Is there a solution to support IOPS constraints on a specific rbd volume from Ceph side? I konw Nautilus may support, but i am a bit hesitate to upgrade. I would like to give Nautilus more time to become more mature. > > thnaks a lot in advance, There is no current support backend Ceph QoS (it's in development) -- just frontend QoS throttles which OpenStack handles via QEMU. > samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Jason

1 0

RBD volume QoS support
by huxiaoyu＠horebdata.cn 25 Aug '20

25 Aug '20

Dear Ceph folks, I am running Openstack Queens to host a variety of Apps, with ceph backend storage Luminous 12.2.13. Is there a solution to support IOPS constraints on a specific rbd volume from Ceph side? I konw Nautilus may support, but i am a bit hesitate to upgrade. I would like to give Nautilus more time to become more mature. thnaks a lot in advance, samuel huxiaoyu(a)horebdata.cn

1 0

Cluster experiencing complete operational failure, various cephx authentication errors
by Mathijs Smit 25 Aug '20

25 Aug '20

Hi everyone, I have a serious problem which currently exists of my entire Ceph no longer being able to provide service. As if yesterday I added 10 OSD's total 2 per node, the rebalance started and took some IO but seemed to be doing its work. This morning the cluster was still processing the rebalance and taking so much IO that nearly all OSD's where marked as "slow ops" and from there everything went wrong. As attempt to clear as much IO for de rebalance I stoped all the clients and waited for the rebalance to finish. After it finished the cluster remained extremely slow and unusable. Whilst trying to debug I restarted several services and nodes trying to find the problem. Now the cluster has entered a state where multiple OSD's remain slow, various OSD's show a "BADAUTHORIZER" message and the mgr on all nodes also has issues "verify_authorizer". I verified all the clocks on all servers and they are sinked to the same NTP service and seem good. Please please please advise as straight 13 hours of debugging got me nowhere. Current version: ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable) Mgr error example: 2020-08-24 20:19:12.865 7f2baf56a700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=12230 2020-08-24 20:19:13.043 7f2bb056c700 0 auth: could not find secret_id=12230 2020-08-24 20:19:13.043 7f2bb056c700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=12230 2020-08-24 20:19:13.210 7f2bb056c700 0 auth: could not find secret_id=12230 2020-08-24 20:19:13.210 7f2bb056c700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=12230 OSD error example: 2020-08-24 19:47:15.777 7f9957d79700 -1 osd.19 41255 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.1510:4 12.a6 12.4b2c82a6 (undecoded) ondisk+retry+read+known_if_redirected+full_force e41119) 2020-08-24 19:47:15.833 7f995c88b700 0 auth: could not find secret_id=12230 2020-08-24 19:47:15.833 7f995c88b700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=12230 2020-08-24 19:47:15.833 7f995c88b700 0 --1- [v2:10.201.1.17:6814/1030299,v1:10.201.1.17:6815/1030299] >> v1:10.201.1.20:6823/1023281 conn(0x55affb8e7c00 0x55b007441000 :6815 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0

2 2

Re: Add OSD with primary on HDD, WAL and DB on SSD
by Tony Liu 25 Aug '20

25 Aug '20

> -----Original Message----- > From: Anthony D'Atri <anthony.datri(a)gmail.com> > Sent: Monday, August 24, 2020 7:30 PM > To: Tony Liu <tonyliu0592(a)hotmail.com> > Subject: Re: [ceph-users] Re: Add OSD with primary on HDD, WAL and DB on > SSD > > Why such small HDDs? Kinda not worth the drive bays and power, instead > of the complexity of putting WAL+DB on a shared SSD, might you have been > able to just buy SSDs and not split? ymmv. 2TB is for testing, it will bump up to 10TB for production. > The limit is a function of the way the DB levels work, it’s not > intentional. > > WAL by default takes a fixed size, like 512 MB or something. > > 64 GB is a reasonable size, it accomodates the WAL and allows space for > DB compaction without overflowing. For each 10TB HDD, what's the recommended DB device size for both DB and WAL? The doc recommends 1% - 4%, meaning 100GB - 400GB for each 10TB HDD. But given the WAL data size and DB data size, I am not sure if that 100GB - 400GB will be used efficiently. > With this commit the situation should be improved, though you don’t > mention what release you’re running > > https://github.com/ceph/ceph/pull/29687 I am using ceph version 15.2.4 octopus (stable). Thanks! Tony > >>> I don't need to create > >>> WAL device, just primary on HDD and DB on SSD, and WAL will be using > >>> DB device cause it's faster. Is that correct? > >> > >> Yes. > >> > >> > >> But be aware that the DB sizes are limited to 3GB, 30GB and 300GB. > >> Anything less than those sizes will have a lot of untilised space, > >> e.g a 20GB device will only utilise 3GB. > > > > I have 1 480GB SSD and 7 2TB HDDs. 7 LVs are created on SSD, each is > > about 64GB, for 7 OSDs. > > > > Since it's shared by DB and WAL, DB will take 30GB and WAL will take > > the rest 34GB. Is that correct? > > > > Is that size of DB and WAL good for 2TB HDD (block store and object > > store cases)? > > > > Could you share a bit more about the intention of such limit? > > > > > > Thanks! > > Tony > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io

1 0

Re: Add OSD with primary on HDD, WAL and DB on SSD
by Lindsay Mathieson 25 Aug '20

25 Aug '20

On 25/08/2020 6:07 am, Tony Liu wrote: > I don't need to create > WAL device, just primary on HDD and DB on SSD, and WAL will be > using DB device cause it's faster. Is that correct? Yes. But be aware that the DB sizes are limited to 3GB, 30GB and 300GB. Anything less than those sizes will have a lot of untilised space, e.g a 20GB device will only utilise 3GB. -- Lindsay

2 1

Re: Add OSD with primary on HDD, WAL and DB on SSD
by ceph＠elchaka.de 24 Aug '20

24 Aug '20

Hi, you could try to use ceph-volume lvm create --data DEV --db DEV.... and inspect the output to learn what is being done. I am not sure about the right Syntax now but you should find related Information via search ... Hth Mehmet Am 23. August 2020 05:52:29 MESZ schrieb Tony Liu <tonyliu0592(a)hotmail.com>: >Hi, > >I am able to add OSD with primary on HDD, WAL and DB on SSD >on dashboard. LVs are created on SSD for OSDs. >How can I achieve the same by CLI? > >I tried with drivegroups, which adds OSD, but not WAL and DB. >Should I handle LV for WAL and DB separately? >Anything I am missing here? > > >Thanks! > >Tony > > >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Add OSD with primary on HDD, WAL and DB on SSD
by Eugen Block 24 Aug '20

24 Aug '20

Hi, if you shared your drivegroup config we might be able to help identify your issue. ;-) The last example in [1] shows the "wal_devices" filter for splitting wal and db. Regards, Eugen [1] https://docs.ceph.com/docs/master/cephadm/drivegroups/#dedicated-wal-db Zitat von Tony Liu <tonyliu0592(a)hotmail.com>: > Hi, > > I am able to add OSD with primary on HDD, WAL and DB on SSD > on dashboard. LVs are created on SSD for OSDs. > How can I achieve the same by CLI? > > I tried with drivegroups, which adds OSD, but not WAL and DB. > Should I handle LV for WAL and DB separately? > Anything I am missing here? > > > Thanks! > > Tony > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Adding OSD
by jcharles＠provectio.fr 24 Aug '20

24 Aug '20

Hello, I just add one new osd in one of my osd hosts. Crush map is updated but nothing happen, the new osd is seen by the cluster, but norebalance occurs. I have to admit I am a bit stressed. I have my current OSD at 70-75% of capacity. May be should I add some osd to other hosts for the cluster start the rebalancing ? Thanks

3 5

Upgrade options and *request for comment
by Ed Kalk 24 Aug '20

24 Aug '20

Hello Ceph Users, We have a Mimic 13.2.10 cluster on Ubuntu 16.04.03 with 6 servers, 36 OSDs. Each OSD has a 2GB memory target via ceph.conf (30 spin disks with HDD type and 2 pools on them (2copy and 3copy pools), 6 SSDs in a flash pool (3copy)) Our intention is to upgrade the code level of CEPH only (Ubuntu remains the same on 16.04.03) and we have the choice of goto Nautilus or goto Octopus. **Does anyone have any warning based on upgrade attempts or advisement for Nautilus or Octopus? Our primary concern is simply code stability post upgrade. -- Thank you for your time, Edward H. Kalk IV Information Technology Dept. Server Specialist Datacenter Virtualization and Storage Systems Socket Telecom, LLC. 2703 Clark Lane Columbia, MO 65202 573-817-0000 or 800-socket3 X218

2 1

Add OSD host with not clean disks
by Tony Liu 24 Aug '20

24 Aug '20

Hi, I am trying to add a OSD host with not clean disks. I added the host. ceph orch host add storage-3 10.6.50.82 Host is added, but the disks on that host are not listed. I assume because they are not clean? I zapped all disks. ceph orch device zap storage-3 /dev/sdb ...... All disks are cleaned successfully. But I still don't see those disks listed. I tried to remove the host and add it back. ceph orch host rm storage-3 ceph orch host add storage-3 10.6.50.82 I still don't see those disks listed. Finally, I restarted mgd, then I see those disks listed. Is this expected or anything I'm missing? Thanks! Tony

1 0

Office Setup And Product Key
by sapohoj267＠qqmimpi.com 24 Aug '20

24 Aug '20

The office is a productivity suite, developed and maintained by one of the biggest companies, market leaders in technology, Microsoft Office. You must already have gotten your answer, Microsoft is not just another company that poses to be developing great software with just a team of 100 members but Microsoft is a big company that needs to keep its reputation and work always on par with the revenue they are getting. This means they need to hire the best people to get the job done and that’s how it works in Microsoft. There is no issue with Microsoft, but the only issue is that Microsoft has been the target for hackers for quite a while now, It is one of the easiest targets anyone can find when it comes to the customer base and data theft. The data Microsoft has stored on their cloud servers is enormous and is really important for most of the people. [url=https://www.officesetup.help]office.com/setup[/url] [url=https://w-ww-office.com/setup]office.com/setup[/url] [url=https://www.help-mcafee.me]mcafee.com/activate[/url] [url=https://w-w-w-mcafee.com/activate]mcafee.com/activate[/url]

1 0

How to change wal block in bluestore?
by xux1217＠gmail.com 24 Aug '20

24 Aug '20

Hi, I want to change my wal device to another device. I search much time and write a shell script, but finally, when I start the osd, the process is crashed. I think something is wrong, please help me check the script. Thanks! ceph version: ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable). The shell script is below: #! /bin/bash id=$1 # osd id vg=$2 # vg name osdPath="/var/lib/ceph/osd/ceph-$id" osdService="ceph-osd(a)${id}.service" blockPath=$(ls -lh ${osdPath}/block|awk '{print $11}') if [ -z "$blockPath" ]; then echo "no block path" exit 1; fi walPath=$(ls -lh ${osdPath}/block.wal|awk '{print $11}') if [ -z "$walPath" ]; then echo "no wal path" exit 2; fi walSize=$(blockdev --getsize64 $walPath) if [ -z "$walSize" ]; then echo "no wal size" exit 3; fi walUUID=$(lvdisplay ${walPath}|grep "LV UUID"|awk '{print $3}') if [ -z "$walUUID" ]; then echo "no wal uuid" exit 3; fi newWalLv="journal-osd${id}" newWalPath="/dev/${vg}/${newWalLv}" lvcreate -n "${newWalLv}" -L${walSize}b $vg || exit 4 newWalUUID=$(lvdisplay ${newWalPath}|grep "LV UUID"|awk '{print $3}') if [ -z "$newWalUUID" ]; then echo "no new wal uuid, new wal path:${newWalPath}" exit 3; fi ceph osd set noout || exit 5 systemctl stop ${osdService} || exit 6 ceph-osd -i $id --flush-journal || exit 7 dd if=${walPath} of=${newWalPath} bs=4M || exit 8 chown -h ceph:ceph $(realpath ${newWalPath}) || exit 9 ceph-bluestore-tool bluefs-bdev-migrate --path ${osdPath} --devs-source ${osdPath}/block.wal --dev-target ${newWalPath} chown -h ceph:ceph ${osdPath}/block.wal || exit 11 # device lvchange --deltag "ceph.wal_device=${walPath}" ${blockPath} || exit 12 lvchange --addtag "ceph.wal_device=${newWalPath}" ${blockPath} || exit 13 # for uuid lvchange --deltag "ceph.wal_uuid=${walUUID}" ${blockPath} || exit 12 lvchange --addtag "ceph.wal_uuid=${newWalUUID}" ${blockPath} || exit 13 # for wal lvs for tag in $(lvs -o lv_tags ${blockPath}|grep ceph|sed 's/,/\n/g') do lvchange --addtag "$tag" ${newWalPath} done lvchange --deltag "ceph.type=block" ${newWalPath} lvchange --addtag "ceph.type=wal" ${newWalPath} systemctl start ceph-osd(a)${id}.service || exit 14 ceph osd unset noout || exit 15

2 1

Don’t Work With The Stressed Mind Rather Use Assignment Help
by Ricky Paul 24 Aug '20

24 Aug '20

What you do when you have the pressure of producing outstanding outcomes? Does your mind feel stressed or tired? Do you feel lethargic for writing your assignment? Well, it is quite natural that you feel exhausted when you have to work under stress. Due to this, you may not pay full attention to assignments. For this, one thing you can do is to place your order for assignment help. Under these services of academic writing, you will receive effective support from professionals and experienced academic writers. They know how to write flawless assignments without losing any single information. If you don’t want to lose your marks, you must think about online academic writing services. This is because if you do work on a stressed mind, you can’t generate useful ideas for your assignments and may lose your marks. So, if you are in this situation, you need rest. Take rest and transfer your work to the assignment helper so you will not lose your marks. For more info : https://www.greatassignmenthelp.com

1 0

Don’t Work With The Stressed Mind Rather Use Assignment Help
by Ricky Paul 24 Aug '20

24 Aug '20

What you do when you have the pressure of producing outstanding outcomes? Does your mind feel stressed or tired? Do you feel lethargic for writing your assignment? Well, it is quite natural that you feel exhausted when you have to work under stress. Due to this, you may not pay full attention to assignments. For this, one thing you can do is to place your order for assignment help. Under these services of academic writing, you will receive effective support from professionals and experienced academic writers. They know how to write flawless assignments without losing any single information. If you don’t want to lose your marks, you must think about online academic writing services. This is because if you do work on a stressed mind, you can’t generate useful ideas for your assignments and may lose your marks. So, if you are in this situation, you need rest. Take rest and transfer your work to the assignment helper so you will not lose your marks. For more info : https://www.greatassignmenthelp.com

1 0