September 2020 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

Remapped PGs
by David Orman 03 Mar '21

03 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

Re: mds lost very frequently
by Stefan Kooman 25 Feb '21

25 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

Default data pool in CEPH
by Gabriel Medve 25 Feb '21

25 Feb '21

Hi, I have a CEPH 15.2.4 running in a docker. How to configure for use a specific data pool? i try put the follow line in the ceph.conf but the changes not working. . [client.myclient] rbd default data pool = Mydatapool I need it to configure for erasure pool with cloudstack Can help me? , where is the ceph conf we i need configure? Thanks. -- Untitled Document

4 6

Re: NFS Ganesha NFSv3
by Gabriel Medve 25 Feb '21

25 Feb '21

Hi Thanks for the reply. cephadm runs ceph containers automatically. How to set privileged mode in ceph container? -- > El 23/9/20 a las 13:24, Daniel Gryniewicz escribió: >> NFSv3 needs privileges to connect to the portmapper. Try running >> your docker container in privileged mode, and see if that helps. >> >> Daniel >> >> On 9/23/20 11:42 AM, Gabriel Medve wrote: >>> Hi, >>> >>> I have a CEPH 15.2.5 running in a docker , i configure nfs ganesha >>> with nfs version 3 but i can not mount it. >>> If configure ganesha with nfs version 4 i can mounted without >>> problems but i need the version 3 . >>> >>> The error is mount.nfs: Protocol not supported >>> >>> Can help me? >>> >>> Thanks. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- > Untitled Document

2 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

v15.2.4 Octopus released
by David Galloway 17 Dec '20

17 Dec '20

We're happy to announce the fourth bugfix release in the Octopus series. In addition to a security fix in RGW, this release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-4-octopus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * Cephadm: There were a lot of small usability improvements and bug fixes: * Grafana when deployed by Cephadm now binds to all network interfaces. * `cephadm check-host` now prints all detected problems at once. * Cephadm now calls `ceph dashboard set-grafana-api-ssl-verify false` when generating an SSL certificate for Grafana. * The Alertmanager is now correctly pointed to the Ceph Dashboard * `cephadm adopt` now supports adopting an Alertmanager * `ceph orch ps` now supports filtering by service name * `ceph orch host ls` now marks hosts as offline, if they are not accessible. * Cephadm can now deploy NFS Ganesha services. For example, to deploy NFS with a service id of mynfs, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:: ceph orch apply nfs mynfs nfs-ganesha nfs-ns * Cephadm: `ceph orch ls --export` now returns all service specifications in yaml representation that is consumable by `ceph orch apply`. In addition, the commands `orch ps` and `orch ls` now support `--format yaml` and `--format json-pretty`. * Cephadm: `ceph orch apply osd` supports a `--preview` flag that prints a preview of the OSD specification before deploying OSDs. This makes it possible to verify that the specification is correct, before applying it. * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, and `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * RBD: The name of the rbd pool object that is used to store rbd trash purge schedule is changed from "rbd_trash_trash_purge_schedule" to "rbd_trash_purge_schedule". Users that have already started using `rbd trash purge schedule` functionality and have per pool or namespace schedules configured should copy "rbd_trash_trash_purge_schedule" object to "rbd_trash_purge_schedule" before the upgrade and remove "rbd_trash_purge_schedule" using the following commands in every RBD pool and namespace where a trash purge schedule was previously configured:: rados -p <pool-name> [-N namespace] cp rbd_trash_trash_purge_schedule rbd_trash_purge_schedule rados -p <pool-name> [-N namespace] rm rbd_trash_trash_purge_schedule or use any other convenient way to restore the schedule after the upgrade. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7447c15c6ff58d7fce91843b705a268a1917325c -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

4 5

multiple OSD crash, unfound objects
by Michael Thomas 16 Dec '20

16 Dec '20

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects. The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem). I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg…. I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects. I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below). The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours. My questions are: * Is there a way to force the cluster to query the possible locations sooner? * Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes? --Mike ceph -s: cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops. services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs task status: scrub status: mds.ceph4: idle data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................]

4 34

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

block.db/block.wal device performance dropped after upgrade to 14.2.10
by Vladimir Prokofev 04 Dec '20

04 Dec '20

Good day, cephers! We've recently upgraded our cluster from 14.2.8 to 14.2.10 release, also performing full system packages upgrade(Ubuntu 18.04 LTS). After that performance significantly dropped, main reason beeing that journal SSDs are now have no merges, huge queues, and increased latency. There's a few screenshots in attachments. This is for an SSD journal that supports block.db/block.wal for 3 spinning OSDs, and it looks like this for all our SSD block.db/wal devices across all nodes. Any ideas what may cause that? Maybe I've missed something important in release notes?

6 20

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Advice on SSD choices for WAL/DB?
by Andrei Mikhailovsky 27 Nov '20

27 Nov '20

Hello, We are planning to perform a small upgrade to our cluster and slowly start adding 12TB SATA HDD drives. We need to accommodate for additional SSD WAL/DB requirements as well. Currently we are considering the following: HDD Drives - Seagate EXOS 12TB SSD Drives for WAL/DB - Intel D3 S4510 960GB or Intel D3 S4610 960GB Our cluster isn't hosting any IO intensive DBs nor IO hungry VMs such as Exchange, MSSQL, etc. From the documentation that I've read the recommended size for DB is between 1% and 4% of the size of the osd. Would 2% figure be sufficient enough (so around 240GB DB size for each 12TB osd?) Also, from your experience, which is a better model for the SSD DB/WAL? Would Intel S4510 be sufficient enough for our purpose or would the S4610 be a much better choice? Are there any other cost effective performance to consider instead of the above models? The same question to the HDD. Any other drives we should consider instead of the Seagate EXOS series? Thanks for you help and suggestions. Andrei

4 4

OSD memory leak?
by Frank Schilder 16 Nov '20

16 Nov '20

Hi all, on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load. What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats: Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } } Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Am I looking at a memory leak here or are these heap stats expected? I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 25

NoSuchKey on key that is visible in s3 list/radosgw bk
by Mariusz Gronczewski 06 Nov '20

06 Nov '20

Hi, I've got a problem on Octopus (15.2.3, debian packages) install, bucket S3 index shows a file: s3cmd ls s3://upvid/255/38355 --recursive 2020-07-27 17:48 50584342 s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 radosgw-admin bi list also shows it { "type": "plain", "idx": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "entry": { "name": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "instance": "", "ver": { "pool": 11, "epoch": 853842 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 50584342, "mtime": "2020-07-27T17:48:27.203008Z", "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", "storage_class": "", "owner": "filmweb-app", "owner_display_name": "filmweb app user", "content_type": "", "accounted_size": 50584342, "user_data": "", "appendable": "false" }, "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, but trying to download it via curl (I've set permissions to public0 only gets me <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>upvid</BucketName><RequestId>tx0000000000000000e716d-005f1f14cb-e478a-pl-war1</RequestId><HostId>e478a-pl-war1-pl</HostId></Error> (the actually nonexisting files shows access denied in same context) same with other tools: $ s3cmd get s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 /tmp download: 's3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' -> '/tmp/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' [1 of 1] ERROR: S3 error: 404 (NoSuchKey) cluster health is OK Any ideas what is happening here ? -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: admin(a)efigence.com

3 3

Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Jean-Philippe Méthot 26 Oct '20

26 Oct '20

Hi, We’re upgrading our cluster OSD node per OSD node to Nautilus from Mimic. From some release notes, it was recommended to run the following command to fix stats after an upgrade : ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 However, running that command gives us the following error message: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: In > function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: 53 > : FAILED ceph_assert(r == 0) > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f1a5a823025] > 2: (()+0x25c1ed) [0x7f1a5a8231ed] > 3: (()+0x3c7a4f) [0x55b33537ca4f] > 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 8: (main()+0x10b3) [0x55b335187493] > 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 10: (()+0x1f9b5f) [0x55b3351aeb5f] > 2020-09-10 14:40:25.873 7f1a6467eec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: 53: FAILED ceph_assert(r == 0) > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f1a5a823025] > 2: (()+0x25c1ed) [0x7f1a5a8231ed] > 3: (()+0x3c7a4f) [0x55b33537ca4f] > 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 8: (main()+0x10b3) [0x55b335187493] > 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 10: (()+0x1f9b5f) [0x55b3351aeb5f] > *** Caught signal (Aborted) ** > in thread 7f1a6467eec0 thread_name:ceph-bluestore- > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (()+0xf630) [0x7f1a58cf0630] > 2: (gsignal()+0x37) [0x7f1a574be387] > 3: (abort()+0x148) [0x7f1a574bfa78] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f1a5a823074] > 5: (()+0x25c1ed) [0x7f1a5a8231ed] > 6: (()+0x3c7a4f) [0x55b33537ca4f] > 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 11: (main()+0x10b3) [0x55b335187493] > 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 13: (()+0x1f9b5f) [0x55b3351aeb5f] > 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal (Aborted) ** > in thread 7f1a6467eec0 thread_name:ceph-bluestore- > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (()+0xf630) [0x7f1a58cf0630] > 2: (gsignal()+0x37) [0x7f1a574be387] > 3: (abort()+0x148) [0x7f1a574bfa78] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f1a5a823074] > 5: (()+0x25c1ed) [0x7f1a5a8231ed] > 6: (()+0x3c7a4f) [0x55b33537ca4f] > 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 11: (main()+0x10b3) [0x55b335187493] > 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 13: (()+0x1f9b5f) [0x55b3351aeb5f] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. What could be the source of this error? I haven’t found much of anything about it online. Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. 4414-4416 Louis B Mayer Laval, QC, H7P 0G1, Canada TEL : +1.514.802.1644 - Poste : 2644 FAX : +1.514.612.0678 CA/US : 1.855.774.4678 FR : 01 76 60 41 43 UK : 0808 189 0423

3 7

Re: Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Igor Fedotov 26 Oct '20

26 Oct '20

On 9/25/2020 6:07 PM, Saber(a)PlanetHoster.info wrote: > Hi Igor, > > The only thing abnormal about this osdstore is that it was created by > Mimic 13.2.8 and I can see that the OSDs size of this osdstore are not > the same as the others in the cluster (while they should be exactly > the same size). > > Can it be https://tracker.ceph.com/issues/39151 ? hmm, may be... Did you change H/W at some point for this OSD's node as it happened in the ticket? And it's still unclear to me if the issue is reproducible for you. Could you please also run fsck (at first) and then repair for this OSD and collect log(s). Thanks, Igor > > Thanks! > Saber > CTO @PlanetHoster > >> On Sep 25, 2020, at 5:46 AM, Igor Fedotov <ifedotov(a)suse.de >> <mailto:ifedotov@suse.de>> wrote: >> >> Hi Saber, >> >> I don't think this is related. New assertion happens along the write >> path while the original one occurred on allocator shutdown. >> >> >> Unfortunately there are not much information to troubleshoot this... >> Are you able to reproduce the case? >> >> >> Thanks, >> >> Igor >> >> On 9/25/2020 4:21 AM, Saber(a)PlanetHoster.info wrote: >>> Hi Igor, >>> >>> We had an osd crash a week after running Nautilus. I have attached >>> the logs, is it related to the same bug? >>> >>> >>> >>> >>> Thanks, >>> Saber >>> CTO @PlanetHoster >>> >>>> On Sep 14, 2020, at 10:22 AM, Igor Fedotov <ifedotov(a)suse.de >>>> <mailto:ifedotov@suse.de>> wrote: >>>> >>>> Thanks! >>>> >>>> Now got the root cause. The fix is on its way... >>>> >>>> Meanwhile you might want to try to workaround the issue via setting >>>> "bluestore_hybrid_alloc_mem_cap" to 0 or using different allocator, >>>> e.g. avl for bluestore_allocator (and optionally for >>>> bluefs_allocator too). >>>> >>>> >>>> Hope this helps, >>>> >>>> Igor. >>>> >>>> >>>> >>>> On 9/14/2020 5:02 PM, Jean-Philippe Méthot wrote: >>>>> Alright, here’s the full log file. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Jean-Philippe Méthot >>>>> Senior Openstack system administrator >>>>> Administrateur système Openstack sénior >>>>> PlanetHoster inc. >>>>> 4414-4416 Louis B Mayer >>>>> Laval, QC, H7P 0G1, Canada >>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>> FAX : +1.514.612.0678 >>>>> CA/US : 1.855.774.4678 >>>>> FR : 01 76 60 41 43 >>>>> UK : 0808 189 0423 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Le 14 sept. 2020 à 06:49, Igor Fedotov <ifedotov(a)suse.de >>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>> >>>>>> Well, I can see duplicate admin socket command >>>>>> registration/de-registration (and the second de-registration >>>>>> asserts) but don't understand how this could happen. >>>>>> >>>>>> Would you share the full log, please? >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Igor >>>>>> >>>>>> On 9/11/2020 7:26 PM, Jean-Philippe Méthot wrote: >>>>>>> Here’s the out file, as requested. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Jean-Philippe Méthot >>>>>>> Senior Openstack system administrator >>>>>>> Administrateur système Openstack sénior >>>>>>> PlanetHoster inc. >>>>>>> 4414-4416 Louis B Mayer >>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>> FAX : +1.514.612.0678 >>>>>>> CA/US : 1.855.774.4678 >>>>>>> FR : 01 76 60 41 43 >>>>>>> UK : 0808 189 0423 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Le 11 sept. 2020 à 10:38, Igor Fedotov <ifedotov(a)suse.de >>>>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>>>> >>>>>>>> Could you please run: >>>>>>>> >>>>>>>> CEPH_ARGS="--log-file log --debug-asok 5" ceph-bluestore-tool >>>>>>>> repair --path <...> ; cat log | grep asok > out >>>>>>>> >>>>>>>> and share 'out' file. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Igor >>>>>>>> >>>>>>>> On 9/11/2020 5:15 PM, Jean-Philippe Méthot wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> We’re upgrading our cluster OSD node per OSD node to Nautilus >>>>>>>>> from Mimic. From some release notes, it was recommended to run >>>>>>>>> the following command to fix stats after an upgrade : >>>>>>>>> >>>>>>>>> ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 >>>>>>>>> >>>>>>>>> However, running that command gives us the following error >>>>>>>>> message: >>>>>>>>> >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: In >>>>>>>>>> function 'virtual Allocator::SocketHook::~SocketHook()' >>>>>>>>>> thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: 53 >>>>>>>>>> : FAILED ceph_assert(r == 0) >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> 2020-09-10 14:40:25.873 7f1a6467eec0 -1 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: In function 'virtual >>>>>>>>>> Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 >>>>>>>>>> time 2020-09-10 14:40:25.872353 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: 53: FAILED ceph_assert(r == 0) >>>>>>>>>> >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> *** Caught signal (Aborted) ** >>>>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal >>>>>>>>>> (Aborted) ** >>>>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>>>> >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS >>>>>>>>>> <executable>` is needed to interpret this. >>>>>>>>> >>>>>>>>> What could be the source of this error? I haven’t found much >>>>>>>>> of anything about it online. >>>>>>>>> >>>>>>>>> >>>>>>>>> Jean-Philippe Méthot >>>>>>>>> Senior Openstack system administrator >>>>>>>>> Administrateur système Openstack sénior >>>>>>>>> PlanetHoster inc. >>>>>>>>> 4414-4416 Louis B Mayer >>>>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>>>> FAX : +1.514.612.0678 >>>>>>>>> CA/US : 1.855.774.4678 >>>>>>>>> FR : 01 76 60 41 43 >>>>>>>>> UK : 0808 189 0423 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> <mailto:ceph-users@ceph.io> >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> <mailto:ceph-users-leave@ceph.io> >>>>>>> >>>>> >>> >

2 1

pool pgp_num not updated
by norman 07 Oct '20

07 Oct '20

Hi guys, When I update the pg_num of a pool, I found it not worked(no rebalanced), anyone know the reason? Pool's info: pool 21 'openstack-volumes-rs' replicated size 3 min_size 2 crush_rule 21 object_hash rjenkins pg_num 1024 pgp_num 512 pgp_num_target 1024 autoscale_mode warn last_change 85103 lfor 82044/82044/82044 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~1e6,1e8~300,4e9~18,502~3f,542~11,554~1a,56f~1d7] pool 22 'openstack-vms-rs' replicated size 3 min_size 2 crush_rule 22 object_hash rjenkins pg_num 512 pgp_num 512 pg_num_target 256 pgp_num_target 256 autoscale_mode warn last_change 84769 lfor 0/0/55294 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd The pgp_num_target is set, but pgp_num not set. I have scale out new OSDs and is backfilling before setting the value, is it the reason?

3 2

S3 multipart upload in Ceph 12.2.11 Luminous
by Eugeniy Khvastunov 05 Oct '20

05 Oct '20

Hi ceph-users, Looks like we faced a broken S3 multipart upload in Ceph 12.2.11 Luminous (MCP 2019.2.0) during thanos setup. The request to initialize the multipart fails. S3cmd log: DEBUG: Sending request method_string='POST', uri='/staging-prometheus-k8s-metrics/test4/dd-50M.rnd?uploads', headers={'x-amz-meta-s3cmd-attrs': 'uid:1011/gname: yevhen.kh/uname:yevhen.kh/gid:1011/mode:33188/mtime:1600198080/atime:1600198530/md5:730ee931c6058cad5bd4d87a6c8f3178/ctime:1600198363', 'content-type': 'application/octet-stream', 'Authorization': 'AWS ZLMVD7JZZ8V09VVK66C3:YX5Oe+Ig0FPDbRpIYO2m1WHITOE=', 'x-amz-date': 'Tue, 15 Sep 2020 19:38:33 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0 bytes) DEBUG: Response: {'status': 400, 'headers': {'content-length': '253', 'accept-ranges': 'bytes', 'server': 'nginx', 'connection': 'keep-alive', 'x-amz-request-id': 'tx0000000000000001e019a-005f611839-deb80f-default', 'date': 'Tue, 15 Sep 2020 19:38:37 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad Request', 'data': '<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>staging-prometheus-k8s-metrics</BucketName><RequestId>tx0000000000000001e019a-005f611839-deb80f-default</RequestId><HostId>deb80f-default-default</HostId></Error>'} DEBUG: ConnMan.put(): connection put back to pool ( https://s3.cloud.f.in.company.com:8080#1) DEBUG: S3Error: 400 (Bad Request) DEBUG: HttpHeader: content-length: 253 DEBUG: HttpHeader: accept-ranges: bytes DEBUG: HttpHeader: server: nginx DEBUG: HttpHeader: connection: keep-alive DEBUG: HttpHeader: x-amz-request-id: tx0000000000000001e019a-005f611839-deb80f-default DEBUG: HttpHeader: date: Tue, 15 Sep 2020 19:38:37 GMT DEBUG: HttpHeader: content-type: application/xml DEBUG: ErrorXML: Code: 'InvalidArgument' DEBUG: ErrorXML: BucketName: 'staging-prometheus-k8s-metrics' DEBUG: ErrorXML: RequestId: 'tx0000000000000001e019a-005f611839-deb80f-default' DEBUG: ErrorXML: HostId: 'deb80f-default-default' ERROR: S3 error: 400 (InvalidArgument) CEPH log: > 2020-09-10 08:26:26.238814 7f7482fdd700 1 -- 10.14.2.66:0/3902939317 <== > mon.0 10.14.2.66:6789/0 19 ==== pool_op_reply(tid 1290 (22) Invalid > argument v22601) v1 ==== 43+0+0 (3815426539 0 0) 0x55faeb4e4a80 con > 0x55faeabc8800 > 2020-09-10 08:26:26.241214 7f74607de700 2 req 484:1.595459:s3:POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:completing > 2020-09-10 08:26:26.241389 7f74607de700 2 req 484:1.595632:s3:POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:op > status=-22 > 2020-09-10 08:26:26.241414 7f74607de700 2 req 484:1.595658:s3:POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:http > status=400 > 2020-09-10 08:26:26.241420 7f74607de700 1 ====== req done > req=0x7f74607d8140 op status=-22 http_status=400 ====== > 2020-09-10 08:26:26.241443 7f74607de700 20 process_request() returned -22 > 2020-09-10 08:26:26.241492 7f74607de700 1 civetweb: 0x55faeaedf000: > 10.14.5.203 - - [10/Sep/2020:08:26:24 +0000] "POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001?uploads= > HTTP/1.1" 400 0 - MinIO (linux; amd64) minio-go/v7.0.2 > thanos-sidecar/0.15.0 (go1.14.2) > The bucket exists. The issue is repeated on the newly created bucket and on the existing one. If s3cmd runs with `--multipart-chunk-size-mb=` option and the value is larger than the file size(for example, if the file is 50 mb, set --multipart-chunk-size-mb=200 mb) then the file uploads. Example: > s3cmd -c /root/f-s3-thanos.cfg put /root/dd-50M.rnd > s3://staging-prometheus-k8s-metrics/test4/dd-50M.rnd -d > --multipart-chunk-size-mb=200 > If s3cmd runs with `--disable-multipart` that works as well. Have you faced such problems? I would be grateful for your advice. -- wbr. Eugeniy Khvastunov, System administrator. [FMGH-UANIC] http://blog.unlimite.net

2 2

Feedback for proof of concept OSD Node
by Ignacio Ocampo 05 Oct '20

05 Oct '20

Hi All :), I would like to get your feedback about the components below to build a PoC OSD Node (I will build 3 of these). SSD for OS. NVMe for cache. HDD for storage. The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories. [image: image.png] Thanks for your feedback! -- Ignacio Ocampo

6 12

hdd pg's migrating when converting ssd class osd's
by Marc Roos 01 Oct '20

01 Oct '20

I have been converting ssd's osd's to dmcrypt, and I have noticed that pg's of pools are migrated that should be (and are?) on hdd class. On a healthy ok cluster I am getting, when I set the crush reweight to 0.0 of a ssd osd this: 17.35 10415 0 0 9907 0 36001743890 0 0 3045 3045 active+remapped+backfilling 2020-09-27 12:55:49.093054 83758'20725398 83758:100379720 [8,14,23] 8 [3,14,23] 3 83636'20718129 2020-09-27 00:58:07.098096 83300'20689151 2020-09-24 21:42:07.385360 0 However osds 3,14,23,8 are all hdd osd's Since this is a cluster from Kraken/Luminous, I am not sure if the device class of the replicated_ruleset[1] was set when the pool 17 was created. Weird thing is that all pg's of this pool seem to be on hdd osd[2] Q. How can I display the definition of 'crush_rule 0' at the time of the pool creation? (To be sure it had already this device class hdd configured) [1] [@~]# ceph osd pool ls detail | grep 'pool 17' pool 17 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 83712 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd [@~]# ceph osd crush rule dump replicated_ruleset { "rule_id": 0, "rule_name": "replicated_ruleset", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } [2] [@~]# for osd in `ceph pg dump pgs| grep '^17' | awk '{print $17" "$19}' | grep -oE '[0-9]{1,2}'| sort -u -n`; do ceph osd crush get-device-class osd.$osd ; done | sort -u dumped pgs hdd

5 19

Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs
by Janne Johansson 30 Sep '20

30 Sep '20

Den mån 21 sep. 2020 kl 16:15 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > When I create a new encrypted osd with ceph volume[1] > > Q4: Where is this luks passphrase stored? > I think the OSD asks the mon for it after auth:ing, so "in the mon DBs" somewhere. -- May the most significant bit of your life be positive.

4 3

objects misplaced jumps up at 5%
by Jake Grimmett 30 Sep '20

30 Sep '20

Dear All, After adding 10 new nodes, each with 10 OSDs to a cluster, we are unable to get "objects misplaced" back to zero. The cluster successfully re-balanced from ~35% to 5% misplaced, however every time "objects misplaced" drops below 5%, a number of pgs start to backfill, increasing the "objects misplaced" to 5.1% I do not believe the balancer is active: [root@ceph7 ceph]# ceph balancer status { "last_optimize_duration": "", "plans": [], "mode": "upmap", "active": false, "optimize_result": "", "last_optimize_started": "" } The cluster has now been stuck at ~5% misplaced for a couple of weeks. The recovery is using ~1GiB/s bandwidth, and is preventing any scrubs. The cluster contains 2.6PB of cephfs, that is still read/write usable. Cluster originally had 10 nodes, each with 45 8TB drives. The new nodes have 10 x 16TB drives. To show the cluster before and immediately after an "episode" *************************************************** [root@ceph7 ceph]# ceph -s cluster: id: 36ed7113-080c-49b8-80e2-4947cc456f2a health: HEALTH_WARN 7 nearfull osd(s) 2 pool(s) nearfull Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull 16372 pgs not deep-scrubbed in time 16372 pgs not scrubbed in time 1/3 mons down, quorum ceph1b,ceph3b services: mon: 3 daemons, quorum ceph1b,ceph3b (age 6d), out of quorum: ceph2b mgr: ceph3(active, since 3d), standbys: ceph1 mds: cephfs:1 {0=ceph1=up:active} 1 up:standby-replay osd: 554 osds: 554 up (since 4d), 554 in (since 5w); 848 remapped pgs task status: scrub status: mds.ceph1: idle mds.ceph2: idle data: pools: 3 pools, 16417 pgs objects: 937.39M objects, 2.6 PiB usage: 3.2 PiB used, 1.4 PiB / 4.6 PiB avail pgs: 467620187/9352502650 objects misplaced (5.000%) 7893 active+clean 7294 active+clean+snaptrim_wait 785 active+remapped+backfill_wait 382 active+clean+snaptrim 52 active+remapped+backfilling 11 active+remapped+backfill_wait+backfill_toofull io: client: 129 KiB/s rd, 82 MiB/s wr, 3 op/s rd, 53 op/s wr recovery: 1.1 GiB/s, 364 objects/s *************************************************** and then seconds later: *************************************************** [root@ceph7 ceph]# ceph -s cluster: id: 36ed7113-080c-49b8-80e2-4947cc456f2a health: HEALTH_WARN 7 nearfull osd(s) 2 pool(s) nearfull Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull 16372 pgs not deep-scrubbed in time 16372 pgs not scrubbed in time 1/3 mons down, quorum ceph1b,ceph3b services: mon: 3 daemons, quorum ceph1b,ceph3b (age 6d), out of quorum: ceph2b mgr: ceph3(active, since 3d), standbys: ceph1 mds: cephfs:1 {0=ceph1=up:active} 1 up:standby-replay osd: 554 osds: 554 up (since 5d), 554 in (since 5w); 854 remapped pgs task status: scrub status: mds.ceph1: idle mds.ceph2: idle data: pools: 3 pools, 16417 pgs objects: 937.40M objects, 2.6 PiB usage: 3.2 PiB used, 1.4 PiB / 4.6 PiB avail pgs: 470821753/9352518510 objects misplaced (5.034%) 7892 active+clean 7290 active+clean+snaptrim_wait 791 active+remapped+backfill_wait 381 active+clean+snaptrim 52 active+remapped+backfilling 11 active+remapped+backfill_wait+backfill_toofull io: client: 155 KiB/s rd, 125 MiB/s wr, 2 op/s rd, 53 op/s wr recovery: 969 MiB/s, 330 objects/s *************************************************** If it helps, I've tried capturing 1/5 debug logs from an OSD. Not sure, but I think this is the way to follow a thread handling one pg as it decides to rebalance: [root@ceph7 ceph]# grep 7f2e569e9700 ceph-osd.312.log | less 2020-09-24 14:44:36.844 7f2e569e9700 1 osd.312 pg_epoch: 106808 pg[5.157ds0( v 106803'6043528 (103919'6040524,106803'6043528] local-lis/les=102671/102672 n=56293 ec=85890/1818 lis/c 102671/10 2671 les/c/f 102672/102672/0 106808/106808/106808) [148,508,398,457,256,533,137,469,357,306]p148(0) r=-1 lpr=106808 pi=[102671,106808)/1 luod=0'0 crt=106803'6043528 lcod 106801'6043526 active mbc={} ps=104] start_peering_interval up [312,424,369,461,546,525,498,169,251,127] -> [148,508,398,457,256,533,137,469,357,306], acting [312,424,369,461,546,525,498,169,251,127] -> [148,508,39 8,457,256,533,137,469,357,306], acting_primary 312(0) -> 148, up_primary 312(0) -> 148, role 0 -> -1, features acting 4611087854031667199 upacting 4611087854031667199 2020-09-24 14:44:36.847 7f2e569e9700 1 osd.312 pg_epoch: 106808 pg[5.157ds0( v 106803'6043528 (103919'6040524,106803'6043528] local-lis/les=102671/102672 n=56293 ec=85890/1818 lis/c 102671/102671 les/c/f 102672/102672/0 106808/106808/106808) [148,508,398,457,256,533,137,469,357,306]p148(0) r=-1 lpr=106808 pi=[102671,106808)/1 crt=106803'6043528 lcod 106801'6043526 unknown NOTIFY mbc={} ps=104] state<Start>: transitioning to Stray 2020-09-24 14:44:37.792 7f2e569e9700 1 osd.312 pg_epoch: 106809 pg[5.157ds0( v 106803'6043528 (103919'6040524,106803'6043528] local-lis/les=102671/102672 n=56293 ec=85890/1818 lis/c 102671/102671 les/c/f 102672/102672/0 106808/106809/106809) [148,508,398,457,256,533,137,469,357,306]/[312,424,369,461,546,525,498,169,251,127]p312(0) r=0 lpr=106809 pi=[102671,106809)/1 crt=106803'6043528 lcod 106801'6043526 mlcod 0'0 remapped NOTIFY mbc={} ps=104] start_peering_interval up [148,508,398,457,256,533,137,469,357,306] -> [148,508,398,457,256,533,137,469,357,306], acting [148,508,398,457,256,533,137,469,357,306] -> [312,424,369,461,546,525,498,169,251,127], acting_primary 148(0) -> 312, up_primary 148(0) -> 148, role -1 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-09-24 14:44:37.793 7f2e569e9700 1 osd.312 pg_epoch: 106809 pg[5.157ds0( v 106803'6043528 (103919'6040524,106803'6043528] local-lis/les=102671/102672 n=56293 ec=85890/1818 lis/c 102671/102671 les/c/f 102672/102672/0 106808/106809/106809) [148,508,398,457,256,533,137,469,357,306]/[312,424,369,461,546,525,498,169,251,127]p312(0) r=0 lpr=106809 pi=[102671,106809)/1 crt=106803'6043528 lcod 106801'6043526 mlcod 0'0 remapped mbc={} ps=104] state<Start>: transitioning to Primary 2020-09-24 14:44:38.832 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.137(6) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.861 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.148(0) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.879 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.256(4) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.894 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.306(9) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.902 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.357(8) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.912 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.398(2) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.923 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.457(3) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.931 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.469(7) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.938 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.508(1) from (0'0,0'0] MAX to 106803'6043528 2020-09-24 14:44:38.947 7f2e569e9700 0 log_channel(cluster) log [DBG] : 5.157ds0 starting backfill to osd.533(5) from (0'0,0'0] MAX to 106803'6043528 *************************************************** any advice appreciated, Jake -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.

6 10

RBD huge diff between random vs non-random IOPs - all flash
by tri＠postix.net 30 Sep '20

30 Sep '20

Hi all, I'm trying to troubleshoot an interesting problem with RBD performance for VMs. Tests were done using fio both outside and inside the VMs shows that random read/write is 20-30% slower than bulk read/write at QD=1. However, at QD=16/32/64, random read/write is sometimes 3X faster than bulk read/write. Inside the VMs, tests were done with -direct=1 -sync=1 using libaio. Outside VMs, test were done with -direct=1 -sync=1 with both librbd and libaio. The gap between random and bulk I/O narrows with increasing QD to 128. However, there's always a 20-30% difference with random I/O being faster. Read and write tests show similar results both inside and outside VMs. Typically, the random I/O performance would be less (or much less) than bulk. Any idea as to what I should be looking at? Thanks. Tri Hoang Inside VM ======== At QD=8, randread is around 2.8X read @ 64k --------------------------------------------------------------- tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=8 -direct=1 -sync=1 -rw=randread -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8 fio-3.12 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=762MiB/s][r=12.2k IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1551: Wed Sep 30 08:19:56 2020 read: IOPS=11.9k, BW=743MiB/s (779MB/s)(4096MiB/5515msec) tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=8 -direct=1 -sync=1 -rw=read -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=268MiB/s][r=4289 IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1554: Wed Sep 30 08:22:03 2020 read: IOPS=4374, BW=273MiB/s (287MB/s)(4096MiB/14981msec) At QD=128, randread is around 1.4X read --------------------------------------------------------- tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=128 -direct=1 -sync=1 -rw=randread -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process Jobs: 1 (f=1) read: (groupid=0, jobs=1): err= 0: pid=1548: Wed Sep 30 08:18:59 2020 read: IOPS=23.1k, BW=1441MiB/s (1511MB/s)(4096MiB/2843msec) tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=128 -direct=1 -sync=1 -rw=read -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=974MiB/s][r=15.6k IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1545: Wed Sep 30 08:17:38 2020 read: IOPS=15.9k, BW=997MiB/s (1045MB/s)(4096MiB/4110msec)

2 1

Orchestrator cephadm not setting CRUSH weight on OSD
by Robert Sander 29 Sep '20

29 Sep '20

Hi, I stumbled across an issue where an OSD the gets redeployed has a CRUSH weight of 0 after cephadm finishes. I have created a service definition for the orchestrator to automatically deploy OSDs on SSDs: service_type: osd service_id: SSD_OSDs placement: label: 'osd' data_devices: rotational: 0 size: '100G' These are my steps to reproduce this in a small test cluster running 15.2.4: root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.63994 root default -18 0.81995 rack rack10 -3 0.40996 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 1 ssd 0.09798 osd.1 up 1.00000 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 root@ceph01:~# ceph osd out 1 marked out osd.1. root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.63994 root default -18 0.81995 rack rack10 -3 0.40996 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 1 ssd 0.09798 osd.1 up 0 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 root@ceph01:~# ceph orch osd rm 1 Scheduled OSD(s) for removal 2020-09-10T16:29:58.176991+0200 mgr.ceph02.ouelws [INF] Removing daemon osd.1 from ceph01 2020-09-10T16:30:00.148659+0200 mgr.ceph02.ouelws [INF] Successfully removed OSD <1> on ceph01 root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.54196 root default -18 0.72197 rack rack10 -3 0.31198 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 root@ceph01:~# ceph orch device zap ceph01 /dev/sdc --force INFO:cephadm:/usr/bin/docker:stderr --> Zapping: /dev/sdc INFO:cephadm:/usr/bin/docker:stderr --> Zapping lvm member /dev/sdc. lv_path is /dev/ceph-0d19a151-30b6-459e-936a-488f143e11f6/osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb INFO:cephadm:/usr/bin/docker:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-0d19a151-30b6-459e-936a-488f143e11f6/osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/docker:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/docker:stderr 10+0 records out INFO:cephadm:/usr/bin/docker:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0583658 s, 180 MB/s INFO:cephadm:/usr/bin/docker:stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-0d19a151-30b6-459e-936a-488f143e11f6 INFO:cephadm:/usr/bin/docker:stderr Running command: /usr/sbin/vgremove -v -f ceph-0d19a151-30b6-459e-936a-488f143e11f6 INFO:cephadm:/usr/bin/docker:stderr stderr: Removing ceph--0d19a151--30b6--459e--936a--488f143e11f6-osd--block--d5062900--abe7--413a--9d9a--d1cdda2948eb (253:3) INFO:cephadm:/usr/bin/docker:stderr stderr: Archiving volume group "ceph-0d19a151-30b6-459e-936a-488f143e11f6" metadata (seqno 5). INFO:cephadm:/usr/bin/docker:stderr Releasing logical volume "osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb" INFO:cephadm:/usr/bin/docker:stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-0d19a151-30b6-459e-936a-488f143e11f6" (seqno 6). INFO:cephadm:/usr/bin/docker:stderr stdout: Logical volume "osd-block-d5062900-abe7-413a-9d9a-d1cdda2948eb" successfully removed INFO:cephadm:/usr/bin/docker:stderr stderr: Removing physical volume "/dev/sdc" from volume group "ceph-0d19a151-30b6-459e-936a-488f143e11f6" INFO:cephadm:/usr/bin/docker:stderr stdout: Volume group "ceph-0d19a151-30b6-459e-936a-488f143e11f6" successfully removed INFO:cephadm:/usr/bin/docker:stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync INFO:cephadm:/usr/bin/docker:stderr stderr: 10+0 records in INFO:cephadm:/usr/bin/docker:stderr 10+0 records out INFO:cephadm:/usr/bin/docker:stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.016043 s, 654 MB/s INFO:cephadm:/usr/bin/docker:stderr --> Zapping successful for: <Raw Device: /dev/sdc> 2020-09-10T16:31:15.951617+0200 mgr.ceph02.ouelws [INF] Zap device ceph01:/dev/sdc 2020-09-10T16:31:24.738974+0200 mgr.ceph02.ouelws [INF] Found osd claims for drivegroup SSD_OSDs -> {} 2020-09-10T16:31:24.740489+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph01... 2020-09-10T16:31:31.549897+0200 mgr.ceph02.ouelws [INF] Deploying daemon osd.1 on ceph01 2020-09-10T16:31:33.057061+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph02... 2020-09-10T16:31:33.057373+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph03... 2020-09-10T16:31:33.057519+0200 mgr.ceph02.ouelws [INF] Applying SSD_OSDs on host ceph04... 2020-09-10T16:31:37.569914+0200 mon.ceph01 [INF] osd.1 [v2:10.24.4.128:6810/4173467371,v1:10.24.4.128:6811/4173467371] boot 2020-09-10T16:31:46.531544+0200 mon.ceph01 [INF] Cluster is now healthy root@ceph01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.54196 root default -18 0.72197 rack rack10 -3 0.31198 host ceph01 8 hdd 0.10699 osd.8 up 1.00000 1.00000 9 hdd 0.10699 osd.9 up 1.00000 1.00000 0 ssd 0.09799 osd.0 up 1.00000 1.00000 1 ssd 0 osd.1 up 1.00000 1.00000 -5 0.40999 host ceph02 10 hdd 0.10699 osd.10 up 1.00000 1.00000 11 hdd 0.10699 osd.11 up 1.00000 1.00000 2 ssd 0.09799 osd.2 up 1.00000 1.00000 3 ssd 0.09799 osd.3 up 1.00000 1.00000 -17 0.81999 rack rack11 -7 0.40999 host ceph03 12 hdd 0.10699 osd.12 up 1.00000 1.00000 13 hdd 0.10699 osd.13 up 1.00000 1.00000 4 ssd 0.09799 osd.4 up 1.00000 1.00000 5 ssd 0.09799 osd.5 up 1.00000 1.00000 -9 0.40999 host ceph04 14 hdd 0.10699 osd.14 up 1.00000 1.00000 15 hdd 0.10699 osd.15 up 1.00000 1.00000 6 ssd 0.09799 osd.6 up 1.00000 1.00000 7 ssd 0.09799 osd.7 up 1.00000 1.00000 Why does osd.1 have a weight of 0 now? When the OSDs had been initially deployed with the first ceph orch apply command the weights have been correctly set according to their size. Why is there a difference between this process and an OSD (re-)deployed later on? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 3

Keep having ceph-volume create fail
by Marc Roos 29 Sep '20

29 Sep '20

I have no idea why ceph-volume keeps failing so much. I keep zapping and creating and all of a sudden it works. I am not having pvs or links left in /dev/mapper. I am checking that with lsblk, dmsetup ls --tree and ceph-volume inventory. These are the stdout/err I am having, every time ceph-volume fails it ends with the same stderr output. stdout: Physical volume "/dev/sdh" successfully created. stdout: Volume group "ceph-0fbb2736-5cb1-4f87-aef2-7591fe979360" successfully created stdout: Logical volume "osd-block-3ff3e59c-e752-4560-91b8-b53f38db5c85" created. stderr: got monmap epoch 20 stdout: creating /var/lib/ceph/osd/ceph-21/keyring stdout: creating /var/lib/ceph/osd/ceph-21/lockbox.keyring stderr: Device i7jC8B-0Z5c-z95F-Cewj-3Jz2-60If-jucHBi already exists. stderr: failed to read label for /dev/mapper/i7jC8B-0Z5c-z95F-Cewj-3Jz2-60If-jucHBi: (2) No such file or directory stderr: purged osd.21

1 1

Get Easy Fixes for HP Printer Technical Errors
by pa6724453＠gmail.com 29 Sep '20

29 Sep '20

Our technical experts have many years of troubleshooting experience and great technical skills of handling different types of issues associated to HP printer. HP Printer Support team is technically experienced to identify the main reasons of HP printer not working issue and apply the permanent resolutions to fix this error within a few seconds. Our tech-geeks have the potential to clear user’s doubts and give the best chance to work on your HP printer. https://www.hpprintersupportpro.us/

1 0

123.Hp.Com/Setup | A One Stop Solution For All Your Need
by masonlava77＠gmail.com 29 Sep '20

29 Sep '20

Bought a new printer and ready to set it up? But do you even know how to get started? You can find the printer setup guides on our website and learn how to set up a new HP printer from 123.HP.com/Setup. From unboxing the HP printer to learning how to print and perform other tasks, all the solutions are available on our portal. Still, if you feel like you need more help, we are here to assist you 24/7. You can simply dial our HP printer helpline to get on a call with our printer experts and get the remedy of your problem. https://www.hpprintersupportpro.com/blog/123-hp-com-setup/

1 0

Assignment Help Online
by Ricky Paul 29 Sep '20

29 Sep '20

Assignments help provide professional assignment writers’ assistance to peers so that they can complete their assignments before due dates. Assignments, indeed, are a necessary part of students’ education. For more info : https://www.greatassignmenthelp.com

1 0

Spirit Airlines Phone Number
by emmashiralee＠gmail.com 29 Sep '20

29 Sep '20

You can talk to our travel specialist to find out who will help you with all your queries to dial the <b><a href="https://spiritphonenumber.com/spirit-airlines-phone-number">Spirit Airlines Phone Number</a></b>. We also offer deals such as flight food, medical guidelines, flight timing, ticket cancellation, your flight status, travel packages, affordable tickets, refunds, problems with your seat, ticket booking, flight checks, and many more.

1 0

Ceph RGW Performance
by Dylan Griff 29 Sep '20

29 Sep '20

Hey folks! Just shooting this out there in case someone has some advice. We're just setting up RGW object storage for one of our new Ceph clusters (3 mons, 1072 OSDs, 34 nodes) and doing some benchmarking before letting users on it. We have 10Gb network to our two RGW nodes behind a single ip on haproxy, and some iperf testing shows I can push that much; latencies look okay. However, when using a small cosbench cluster I am unable to get more than ~250Mb of read speed total. If I add more nodes to the cosbench cluster it just spreads out the load evenly with the same cap Same results when running two cosbench clusters from different locations. I don't see any obvious bottlenecks in terms of the RGW server hardware limitations, but here I am asking for assistance so I don't put it past me missing something. I have attached one of my cosbench load files with keys removed, but I get similar results with different numbers of workers, objects, buckets, object sizes, and cosbench drivers. Does anyone have any pointers on what I could find to nail this bottleneck down? Am I wrong in expecting more throughput? Let me know if I can get any other info for you. Cheers, Dylan -- Dylan Griff Senior System Administrator CLE D063 RCS - Systems - University of Victoria

4 4

Doing minor version update of Ceph cluster with ceph-ansible and rolling-update playbook
by andreas.elvers+lists.ceph.io＠lfda.de 28 Sep '20

28 Sep '20

I want to update my mimic cluster to the latest minor version using the rolling-update script of ceph-ansible. The cluster was rolled out with that setup. So as long as ceph_stable_release stays on the current installed version (mimic) the rolling update script will do only a minor update. Is this assumption correct? The documentation (https://docs.ceph.com/projects/ceph-ansible/en/latest/day-2/upgrade.html) is short on this. Thanks! - Andreas

2 1

How OSD encryption affects latency/iops on NVMe, SSD and HDD
by tri＠postix.net 28 Sep '20

28 Sep '20

Hi all, For those who use encryption on your OSDs, what effect do you see on your NVMe, SSD and HDD vs non-encrypted OSDs? I tried to find some info on this subject but there isn't much detail available. From experience, dmcrypt is CPU-bound and becomes a bottleneck when used on very fast NVMe. Using aes-xts, one can only expect around 1600-2000GB/s with 256/512 bit keys. Best, Tri Hoang

3 3

Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early
by Wido den Hollander 28 Sep '20

28 Sep '20

Hi, On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing OSDs randomly drop out of the cluster. Usually it's 2 to 4 OSDs spread out over different nodes. Each node has 16 OSDs and not all the failing OSDs are on the same node. The OSDs are marked as down and all they keep print in their logs: monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-06-04T07:57:17.706529-0400) Looking at their status through the admin socket: { "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", "whoami": 206, "state": "active", "oldest_map": 73697, "newest_map": 75795, "num_pgs": 19 } The message brought me to my own ticket I created 2 years ago: https://tracker.ceph.com/issues/23460 The first thing I've checked is NTP/time. Double, triple check this. All the times are in sync on the cluster. Nothing wrong there. Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. Restarting them brings them back right away and then within 24h some other OSDs will drop out. Has anybody seen this behavior with Octopus as well? Wido

2 1

online Assignment Help
by tillockwatson＠gmail.com 28 Sep '20

28 Sep '20

There are many, but we are the only one who has the largest amount of satisfied clients. Catering higher services to students is our solely main motivation and accomplishment. We have been successful in doing this for the past twelve years and our online assignment help services have gotten approval from the students. We perceive the necessity of the students and committed to essentially fulfil it anyhow. https://www.assignmentachievers.com/service-detail/assignment+help+online

1 0

Assignment Help Experts
by tillockwatson＠gmail.com 28 Sep '20

28 Sep '20

I'm good with writing; love to read, write, edit. I'm a versatile professional researcher, assignment help and academic writer with more than six years of writing experience.

1 0

[orch] redeploying OSDs
by mnaser＠vexxhost.com 27 Sep '20

27 Sep '20

Hi folks: I've been playing around with the new Ceph orchestrator and I've been running into an interesting limitation. I have not found any documented process or combination of things to do which can allow me to 're-adopt' cephadm deployed OSDs. Think use case of reinstalling operating systems, in the old world, we'd just run: `ceph-volume lvm activate` --all and we'll be good to go. Is there any equivalent right now? Thanks Mohammed

1 0

Fwd: [ceph-mgr - 15.2.4 octopus] the cpeh-mgr failed to get the correct status of all PGs
by HAO Xiong 26 Sep '20

26 Sep '20

Hi guys, We recently upgrade the ceph-mgr to 15.2.4, Octopus in our production clusters. The status of the cluster now is as follow: *# ceph versions* *{* * "mon": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 5* * },* * "mgr": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 3* * },* * "osd": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 1933* * },* * "mds": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 14* * },* * "overall": {* * "ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)": 1955* * }* *}* Now we suffered some problems in this cluster: 1. it always took a significant longer time to get the result of `ceph pg dump`. 2. the ceph-exportor might failed to get cluster metrics. 3. sometimes the cluster showed a few inactive/down pgs but recovered very soon. We did a investigation on the ceph-mgr, didn't get the root cause yet. But there are some dispersed clews (I am not sure if they ca help): 1. the ms_dispatch thread is always busy with one core. 2. the msg size is significant larger than 40K. *2020-09-24T14:47:50.216+0000 7f8f811f6700 1 -- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] <== osd.3038 v2:{osd_ip}:6800/384927 431 ==== pg_stats(17 pgs tid 0 v 0) v2 ==== 42153+0+0 (secure 0 0 0) 0x55dae07c1800 con 0x55daf6dde400* 3. get some errors of "Fail to parse JSON result". *2020-09-24T15:47:42.739+0000 7f8f8da0f700 0 [devicehealth ERROR root] Fail to parse JSON result from daemon osd.1292 ()* 4. in the sending channel, we could see lots of faults. *2020-09-24T14:53:17.725+0000 7f8fa866e700 1 -- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] >> v1:{osd_ip}:0/1442957044 conn(0x55db38757400 legacy=0x55db03d8e800 unknown :6801 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (909347879) for more than 900000000 us, fault.* *2020-09-24T14:53:17.725+0000 7f8fa866e700 1 --1- [v2:{mgr_ip}:6800/111,v1:{mgr_ip}:6801/111] >> v1:{osd_ip}:0/1442957044 conn(0x55db38757400 0x55db03d8e800 :6801 s=OPENED pgs=1572189 cs=1 l=1).fault on lossy channel, failing* 5. or the mgr-fin thread would be busy with one core. [image: image.png] and from the perf dump we got: * "finisher-Mgr": { "queue_len": 1359862, "complete_latency": { "avgcount": 14, "sum": 40300.307764855, "avgtime": 2878.593411775 } },* Sorry about these clews are a little messy. Could you have any comments on this? Thanks. Regards, Hao

1 0

Re: rebalancing adapted during rebalancing with new updates?
by Martin Verges 26 Sep '20

26 Sep '20

Without knowing the source code and just from my observations, I would say everytime the osd map changes, the crush/pgmap tries to fix that. However a running backfill is not stopped and only backfill_wait would be reconsidered. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 26. Sept. 2020 um 13:33 Uhr schrieb Marc Roos < M.Roos(a)f1-outsourcing.eu>: > > When I add an osd rebalancing is taking place, lets say ceph relocates > 40 pg's. > > When I add another osd during rebalancing, when ceph has only relocated > 10 pgs and has to do still 30 pgs. > > What happens then: > > 1. Is ceph just finishing the relocation of these 30 pgs and then > calculates how the new environment with the newly added osd should be > relocated and starts relocating that. > > 2. or is ceph is finishing only the relocation of the pg it is currently > doing, and then recalculates immediately how pg's should be distributed > without finishing these 30 pg's it was planning to do. > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

rebalancing adapted during rebalancing with new updates?
by Marc Roos 26 Sep '20

26 Sep '20

When I add an osd rebalancing is taking place, lets say ceph relocates 40 pg's. When I add another osd during rebalancing, when ceph has only relocated 10 pgs and has to do still 30 pgs. What happens then: 1. Is ceph just finishing the relocation of these 30 pgs and then calculates how the new environment with the newly added osd should be relocated and starts relocating that. 2. or is ceph is finishing only the relocation of the pg it is currently doing, and then recalculates immediately how pg's should be distributed without finishing these 30 pg's it was planning to do.

1 0

Technical Assistance
by lirkasilmi＠nedoz.com 26 Sep '20

26 Sep '20

http://garmin-com-express.support/ http://webrootloginn.com/ http://www.123hpcom.de/ https://123hpcomsetup.us/ http://nortonlogins.com/ http://getgarminexpress.com/ https://canoncomijsetup.me/

1 0

how to "undelete" a pool
by Peter Sarossy 26 Sep '20

26 Sep '20

hey folks, I have managed to fat finger a config apply command and accidentally deleted the CRD for one of my pools. The operator went ahead and tried to purge it, but fortunately since it's used by CephFS it was unable to. Redeploying the exact same CRD does not make the operator stop trying to delete it though. Any hints on how to make the operator forget about the deletion request and leave it be? -- Cheers, Peter Sarossy Technical Program Manager Data Center Data Security - Google LLC.

4 6

Re: Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Igor Fedotov 25 Sep '20

25 Sep '20

Hi Saber, I don't think this is related. New assertion happens along the write path while the original one occurred on allocator shutdown. Unfortunately there are not much information to troubleshoot this... Are you able to reproduce the case? Thanks, Igor On 9/25/2020 4:21 AM, Saber(a)PlanetHoster.info wrote: > Hi Igor, > > We had an osd crash a week after running Nautilus. I have attached the > logs, is it related to the same bug? > > > > > Thanks, > Saber > CTO @PlanetHoster > >> On Sep 14, 2020, at 10:22 AM, Igor Fedotov <ifedotov(a)suse.de >> <mailto:ifedotov@suse.de>> wrote: >> >> Thanks! >> >> Now got the root cause. The fix is on its way... >> >> Meanwhile you might want to try to workaround the issue via setting >> "bluestore_hybrid_alloc_mem_cap" to 0 or using different allocator, >> e.g. avl for bluestore_allocator (and optionally for bluefs_allocator >> too). >> >> >> Hope this helps, >> >> Igor. >> >> >> >> On 9/14/2020 5:02 PM, Jean-Philippe Méthot wrote: >>> Alright, here’s the full log file. >>> >>> >>> >>> >>> >>> Jean-Philippe Méthot >>> Senior Openstack system administrator >>> Administrateur système Openstack sénior >>> PlanetHoster inc. >>> 4414-4416 Louis B Mayer >>> Laval, QC, H7P 0G1, Canada >>> TEL : +1.514.802.1644 - Poste : 2644 >>> FAX : +1.514.612.0678 >>> CA/US : 1.855.774.4678 >>> FR : 01 76 60 41 43 >>> UK : 0808 189 0423 >>> >>> >>> >>> >>> >>> >>>> Le 14 sept. 2020 à 06:49, Igor Fedotov <ifedotov(a)suse.de >>>> <mailto:ifedotov@suse.de>> a écrit : >>>> >>>> Well, I can see duplicate admin socket command >>>> registration/de-registration (and the second de-registration >>>> asserts) but don't understand how this could happen. >>>> >>>> Would you share the full log, please? >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 9/11/2020 7:26 PM, Jean-Philippe Méthot wrote: >>>>> Here’s the out file, as requested. >>>>> >>>>> >>>>> >>>>> >>>>> Jean-Philippe Méthot >>>>> Senior Openstack system administrator >>>>> Administrateur système Openstack sénior >>>>> PlanetHoster inc. >>>>> 4414-4416 Louis B Mayer >>>>> Laval, QC, H7P 0G1, Canada >>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>> FAX : +1.514.612.0678 >>>>> CA/US : 1.855.774.4678 >>>>> FR : 01 76 60 41 43 >>>>> UK : 0808 189 0423 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Le 11 sept. 2020 à 10:38, Igor Fedotov <ifedotov(a)suse.de >>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>> >>>>>> Could you please run: >>>>>> >>>>>> CEPH_ARGS="--log-file log --debug-asok 5" ceph-bluestore-tool >>>>>> repair --path <...> ; cat log | grep asok > out >>>>>> >>>>>> and share 'out' file. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Igor >>>>>> >>>>>> On 9/11/2020 5:15 PM, Jean-Philippe Méthot wrote: >>>>>>> Hi, >>>>>>> >>>>>>> We’re upgrading our cluster OSD node per OSD node to Nautilus >>>>>>> from Mimic. From some release notes, it was recommended to run >>>>>>> the following command to fix stats after an upgrade : >>>>>>> >>>>>>> ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 >>>>>>> >>>>>>> However, running that command gives us the following error message: >>>>>>> >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: In >>>>>>>> function 'virtual Allocator::SocketHook::~SocketHook()' thread >>>>>>>> 7f1a6467eec0 time 2020-09-10 14:40:25.872353 >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: 53 >>>>>>>> : FAILED ceph_assert(r == 0) >>>>>>>> ceph version 14.2.11 >>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> 2020-09-10 14:40:25.873 7f1a6467eec0 -1 >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: In function 'virtual >>>>>>>> Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time >>>>>>>> 2020-09-10 14:40:25.872353 >>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>> <http://allocator.cc/>: 53: FAILED ceph_assert(r == 0) >>>>>>>> >>>>>>>> ceph version 14.2.11 >>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> *** Caught signal (Aborted) ** >>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>> ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) >>>>>>>> nautilus (stable) >>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal >>>>>>>> (Aborted) ** >>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>> >>>>>>>> ceph version 14.2.11 >>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] >>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>> [0x55b3352749a1] >>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` >>>>>>>> is needed to interpret this. >>>>>>> >>>>>>> What could be the source of this error? I haven’t found much of >>>>>>> anything about it online. >>>>>>> >>>>>>> >>>>>>> Jean-Philippe Méthot >>>>>>> Senior Openstack system administrator >>>>>>> Administrateur système Openstack sénior >>>>>>> PlanetHoster inc. >>>>>>> 4414-4416 Louis B Mayer >>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>> FAX : +1.514.612.0678 >>>>>>> CA/US : 1.855.774.4678 >>>>>>> FR : 01 76 60 41 43 >>>>>>> UK : 0808 189 0423 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> <mailto:ceph-users@ceph.io> >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>> <mailto:ceph-users-leave@ceph.io> >>>>> >>> >

1 0

Epson printer making odd sound? Arrive at technical support by reaching Epson Customer Service.
by mary smith 25 Sep '20

25 Sep '20

Now and then an issue can emerge when you may hear peculiar sounds originating from the printer because of some tech glitch. In the event that that occurs, at that point you can get the assistance by heading off to the tech help locales or you can call the Epson Customer Service to get the issue settled. https://www.epsonprintersupportpro.net/

1 0

Incapable to see companions on Facebook? Get to Facebook Customer Service Toll Free number.
by mary smith 25 Sep '20

25 Sep '20

The companions on Facebook tells you the number of individuals you're associated with, on the web-based media website. Subsequently, on the off chance that you can't see your companions, at that point this may be because of the helpless web association. On the off chance that you accept that there is some other explanation, at that point you can get to Facebook Customer Service Toll Free Number to know the clear arrangements. https://www.customercare-email.com/facebook-customer-service.html

1 0

NVMe's
by Brent Kennedy 25 Sep '20

25 Sep '20

We currently run a SSD cluster and HDD clusters and are looking at possibly creating a cluster for NVMe storage. For spinners and SSDs, it seemed the max recommended per osd host server was 16 OSDs ( I know it depends on the CPUs and RAM, like 1 cpu core and 2GB memory ). Questions: 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers were bought with 48 cores and 100+ GB of RAM, would this make sense? 2. Should we just raid 5 by groups of NVMe drives instead ( and buy less CPU/RAM )? There is a reluctance to waste even a single drive on raid because redundancy is basically cephs job. 3. The plan was to build this with octopus ( hopefully there are no issues we should know about ). Though I just saw one posted today, but this is a few months off. 4. Any feedback on max OSDs? 5. Right now they run 10Gb everywhere with 80Gb uplinks, I was thinking this would need at least 40Gb links to every node ( the hope is to use these to speed up image processing at the application layer locally in the DC ). I haven't spoken to the Dell engineers yet but my concern with NVMe is that the raid controller would end up being the bottleneck ( next in line after network connectivity ). Regards, -Brent Existing Clusters: Test: Nautilus 14.2.11 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi gateways ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 12 osd servers, 3 mons, 4 gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 3 gateways, 2 iscsi gateways

12 33

Unable to restart OSD assigned to LVM partition on Ceph 15.1.2?
by Matt Larson 24 Sep '20

24 Sep '20

Hi, I recently restarted a storage node for our Ceph cluster and had an issue bringing one of the OSDs back online. This storage node has multiple HDs each as a devoted OSD for a data pool, and a single nVME drive with an LVM partition assigned as an OSD in a metadata pool. After rebooting the host, the OSD using an LVM partition did not restart. When trying to manually start the OSD using systemctl, I can follow the launch of a podman container and see an error message prior to the container shutting down again: Sep 23 14:02:06 X bash[30318]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/boot/cephfs_meta --path /var/lib/ceph/osd/ceph-165 --no-mon-config Sep 23 14:02:06 X bash[30318]: stderr: failed to read label for /dev/boot/cephfs_meta: (2) No such file or directory Sep 23 14:02:06 X bash[30318]: --> RuntimeError: command returned non-zero exit status: 1 1. I can see the existence of the /dev/boot/cephfs_meta symlink to a device ../dm-3 2. `lsblk` shows the lvm partition 'boot-cephfs_meta' under nvme0n1p3 3. `sudo lvscan --all` shows the it as activated: ` ACTIVE '/dev/boot/cephfs_meta' [3.42 TiB] inherit` This is on a CentOS 8 system, with ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) Related issues I have found include: 1. https://github.com/rook/rook/issues/2591 2. https://github.com/rook/rook/issues/3289 There were indicated solutions for these involving installing the LVM2 package, which I completed with `sudo dnf install lvm2`, then tried a restart of the system and restart of the container. This was not able to resolve the problem for LVM-partition based OSD. This LVM-based OSD was initially created with a `ceph-volume` command: `ceph-volume lvm create --bluestore --data /dev/sd<x> --block.db /dev/nvme0n1<partition-nr>` Is there a workaround for this problem where the container process is unable to read the label of the LVM partition and fails to start the OSD? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

1 0

Feature highlight: CephFS network restriction
by Stefan Kooman 24 Sep '20

24 Sep '20

Hi, While reading documentation on CephFS I came accross the "network restriction" feature (supported since Nautilus) [1]. I have never seen a reference to / blog post on this feature and didn't even know it was possible. This might be true for more people, hence this mail to the list :-). So, for improved security you can restrict on ip (ranges) in Ceph (besides / instead) of regular network filtering. I have tested this on Nautilus with IPv6 and it works as documented. Kudos for CephFS team. FYI, Gr. Stefan [1] https://docs.ceph.com/en/latest/cephfs/client-auth/#network-restriction

1 0

RBD quota per namespace
by Eugen Block 24 Sep '20

24 Sep '20

Hi *, I'm curious if this idea [1] of quotas on namespace level for rbd will be implemented. I couldn't find any existing commands in my lab Octopus cluster so I guess it's still just an idea, right? If there's any information I missed please point me to it. Thanks! Eugen [1] https://trello.com/c/Y1n75zpy/396-rbd-per-pool-per-namespace-provision-quot…

3 2

Re: Vitastor, a fast Ceph-like block storage for VMs
by vitalif＠yourcmc.ru 24 Sep '20

24 Sep '20

Hi Roman, Yes, you're right - OSDs list all objects during peering and take the latest full version of each object. Full version is a version that has at least min_size parts for XOR/EC, or any version for replicated setups which is OK because writes are atomic. If there is a newer "committed" ("stabilized") version of the object, but it doesn't have sufficient number of parts, the object is marked as incomplete. By the way, did you receive my reply that I sent directly to your email? - it could've ended up in spam because yourcmc.ru is my own mail server. > Hi Vitaliy, > > Awesome what you did. I like the minimalistic approach focused on > performance very much. A long way you've passed implementing the > whole stack from client to storage. > > Could you please describe peering process and object recovery in more > details? You write in the README that there is no PGLog in your > storage. After a brief glance into osd_ops.h and blockstore_impl.cpp I > see there is a 'version' field. I can assume that every object is > marked with a version and recovery happens taking the whole up-to-date > object, right? > > -- > Roman > > On 2020-09-23 00:44, vitalif(a)yourcmc.ru wrote: > >> Hi! >> After almost a year of development in my spare time I present my own >> software-defined block storage system: Vitastor - https://vitastor.io >> I designed it similar to Ceph in many ways, it also has Pools, PGs, >> OSDs, different coding schemes, rebalancing and so on. However it's >> much simpler and much faster. In a test cluster with SATA SSDs it >> achieved Q1T1 latency of 0.14ms which is especially great compared to >> Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops >> saturation" parallel load benchmark it reached 895k read / 162k write >> iops, compared to Ceph's 480k / 100k on the same hardware, but the >> most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores >> out of 64 on each node and Vitastor was only using 4. >> Of course it's an early pre-release which means that, for example, it >> lacks snapshot support and other useful features. However the base is >> finished - it works and runs QEMU VMs. I like the design and I plan to >> develop it further. >> There are more details in the README file which currently opens from >> the domain https://vitastor.io >> Sorry if it was a bit off-topic, I just thought it could be >> interesting for you :)

1 0

Remove separate WAL device from OSD
by Michael Fladischer 24 Sep '20

24 Sep '20

Hi, Is it possible to remove an existing WAL device from an OSD? I saw that ceph-bluestore-tool has a command bluefs-bdev-migrate, but it's not clear to me if this can only move a WAL device or if it can be used to remove it ... Regards, Michael

4 7

virtual machines crashes after upgrade to octopus
by Lomayani S. Laizer 24 Sep '20

24 Sep '20

Hello, Last week i got time to try debug crashes of these vms Below log includes rados debug which i left last time https://storage.habari.co.tz/index.php/s/AQEJ7tQS7epC4Zn I have observed the following with these settings in openstack and ceph disk_cachemodes="network=writeback" is set in openstack environment and in ceph rbd_cache_policy = writearound is set--- crashes occur disk_cachemodes="network=writeback" is set in openstack environment and in ceph rbd_cache_policy = writeback is set--- no crashes disk_cachemodes="none" is set in openstack environment and in ceph rbd_cache_policy = writearound is set---- no crashes disk_cachemodes="none" is set in openstack environment and in ceph rbd_cache_policy = writeback is set --- crashes occur Is disk_cachemodes="network=writeback" is no longer recommended in octopus because i see it is left out in new documentation for octopus? https://ceph.readthedocs.io/en/latest/rbd/rbd-openstack/ >* > Hello,* >* >* >* > Below is full debug log of 2 minutes before crash of virtual machine.* >* Download from below url* >* >* >* > https://storage.habari.co.tz/index.php/s/31eCwZbOoRTMpcU <https://storage.habari.co.tz/index.php/s/31eCwZbOoRTMpcU>* > >* This log has rbd debug output, but not rados :(* > >* I guess you'll need to try and capture a coredump if you can't get a* >* backtrace.* > >* I'd also suggest opening a tracker in case one of the rbd devs has any* >* ideas on this, or has seen something similar. Without a backtrace or* >* core it will be impossible to definitively identify the issue though.* > +1 to needing the backtrace. I don't see any indications of a problem in that log. >* >* >* >* >* > apport.log* >* >* >* > Wed May 13 09:35:30 2020: host pid 4440 crashed in a separate mount* >* namespace, ignoring* >* >* >* > kernel.log* >* > May 13 09:35:30 compute5 kernel: [123071.373217] fn-radosclient[4485]:* >* segfault at 0 ip 00007f4c8c85d7ed sp 00007f4c66ffc470 error 4 in* >* librbd.so.1.12.0[7f4c8c65a000+5cb000]* >* > May 13 09:35:30 compute5 kernel: [123071.373228] Code: 8d 44 24 08 48 81* >* c3 d8 3e 00 00 49 21 f9 48 c1 e8 30 83 c0 01 48 c1 e0 30 48 89 02 48 8b 03* >* 48 89 04 24 48 8b 34 24 48 21 fe <48> 8b 06 48 89 44 24 08 48 8b 44 24 08* >* 48 8b 0b 48 21 f8 48 39 0c* >* > May 13 09:35:33 compute5 kernel: [123074.832700] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* > May 13 09:35:33 compute5 kernel: [123074.838520] device tap33511c4d-2c* >* left promiscuous mode* >* > May 13 09:35:33 compute5 kernel: [123074.838527] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* >* >* > syslog* >* > compute5 kernel: [123071.373217] fn-radosclient[4485]: segfault at 0 ip* >* 00007f4c8c85d7ed sp 00007f4c66ffc470 error 4 i* >* > n librbd.so.1.12.0[7f4c8c65a000+5cb000]* >* > May 13 09:35:30 compute5 kernel: [123071.373228] Code: 8d 44 24 08 48 81* >* c3 d8 3e 00 00 49 21 f9 48 c1 e8 30 83 c0 01 48 c1 e0 30 48 8* >* > 9 02 48 8b 03 48 89 04 24 48 8b 34 24 48 21 fe <48> 8b 06 48 89 44 24 08* >* 48 8b 44 24 08 48 8b 0b 48 21 f8 48 39 0c* >* > May 13 09:35:30 compute5 libvirtd[1844]: internal error: End of file* >* from qemu monitor* >* > May 13 09:35:33 compute5 systemd-networkd[1326]: tap33511c4d-2c: Link* >* DOWN* >* > May 13 09:35:33 compute5 systemd-networkd[1326]: tap33511c4d-2c: Lost* >* carrier* >* > May 13 09:35:33 compute5 kernel: [123074.832700] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* > May 13 09:35:33 compute5 kernel: [123074.838520] device tap33511c4d-2c* >* left promiscuous mode* >* > May 13 09:35:33 compute5 kernel: [123074.838527] brqa72d845b-e9: port* >* 1(tap33511c4d-2c) entered disabled state* >* > May 13 09:35:33 compute5 networkd-dispatcher[1614]: Failed to request* >* link: No such device* >* >* >* > On Fri, May 8, 2020 at 5:40 AM Brad Hubbard <bhubbard@xxxxxxxxxx> wrote:* >* >>* >* >> On Fri, May 8, 2020 at 12:10 PM Lomayani S. Laizer <lomlaizer@xxxxxxxxx>* >* wrote:* >* >> >* >* >> > Hello,* >* >> > On my side at point of vm crash these are logs below. At the moment* >* my debug is at 10 value. I will rise to 20 for full debug. these crashes* >* are random and so far happens on very busy vms. Downgrading clients in host* >* to Nautilus these crashes disappear* >* >>* >* >> You could try adding debug_rados as well but you may get a very large* >* >> log so keep an eye on things.* >* >>* >* >> >* >* >> > Qemu is not shutting down in general because other vms on the same* >* host continues working* >* >>* >* >> A process can not reliably continue after encountering a segfault so* >* >> the qemu-kvm process must be ending and therefore it should be* >* >> possible to capture a coredump with the right configuration.* >* >>* >* >> In the following example, if you were to search for pid 6060 you would* >* >> find it is no longer running.* >* >> >> > [ 7682.233684] fn-radosclient[6060]: segfault at 2b19 ip* >* 00007f8165cc0a50 sp 00007f81397f6490 error 4 in* >* librbd.so.1.12.0[7f8165ab4000+537000]* >* >>* >* >> Without a backtrace at a minimum it may be very difficult to work out* >* >> what's going on with certainty. If you open a tracker for the issue* >* >> though maybe one of the devs specialising in rbd may have some* >* >> feedback.* >* >>* >* >> >* >* >> > 2020-05-07T13:02:12.121+0300 7f88d57fa700 10 librbd::io::ReadResult:* >* 0x7f88c80bfbf0 finish: got {} for [0,24576] bl 24576* >* >> > 2020-05-07T13:02:12.193+0300 7f88d57fa700 10 librbd::io::ReadResult:* >* 0x7f88c80f9330 finish: C_ObjectReadRequest: r=0* >* >> > 2020-05-07T13:02:12.193+0300 7f88d57fa700 10 librbd::io::ReadResult:* >* 0x7f88c80f9330 finish: got {} for [0,16384] bl 16384* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10 librbd::ImageState:* >* 0x5569b5da9bb0 0x5569b5da9bb0 send_close_unlock* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10 librbd::ImageState:* >* 0x5569b5da9bb0 0x5569b5da9bb0 send_close_unlock* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_block_image_watcher* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 block_notifies* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 5 librbd::Watcher:* >* 0x7f88c400dfe0 block_notifies: blocked_count=1* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_block_image_watcher: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f890ba90500 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_update_watchers* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_shut_down_update_watchers: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_io_queue* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 5* >* librbd::io::ImageRequestWQ: 0x7f88e8001570 shut_down: shut_down: in_flight=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_shut_down_io_queue: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_exclusive_lock* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ExclusiveLock:* >* 0x7f88c4011ba0 shut_down* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 shut_down:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 send_shutdown:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 send_shutdown_release:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10 librbd::ExclusiveLock:* >* 0x7f88c4011ba0 pre_release_lock_handler* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_cancel_op_requests:* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_cancel_op_requests: r=0* >* >> > 2020-05-07T13:02:28.694+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020 send_block_writes:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 5* >* librbd::io::ImageRequestWQ: 0x7f88e8001570 block_writes: 0x5569b5e1ffd0,* >* num=1* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_block_writes: r=0* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020 send_wait_for_ops:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_wait_for_ops:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_invalidate_cache:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 5* >* librbd::io::ObjectDispatcher: 0x5569b5dab700 invalidate_cache:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_invalidate_cache: r=0* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_flush_notifies:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_flush_notifies:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* send_close_object_map:* >* >> > 2020-05-07T13:02:28.698+0300 7f88d4ff9700 10* >* librbd::object_map::UnlockRequest: 0x7f88c807a450 send_unlock:* >* oid=rbd_object_map.2f18f2a67fad72* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::object_map::UnlockRequest: 0x7f88c807a450 handle_unlock: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020* >* handle_close_object_map: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::exclusive_lock::PreReleaseRequest: 0x7f88c80b6020 send_unlock:* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 handle_shutdown_pre_release: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10* >* librbd::managed_lock::ReleaseRequest: 0x7f88c80b68a0 send_unlock:* >* entity=client.58292796, cookie=auto 140225447738256* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 10* >* librbd::managed_lock::ReleaseRequest: 0x7f88c80b68a0 handle_unlock: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ExclusiveLock:* >* 0x7f88c4011ba0 post_release_lock_handler: r=0 shutting_down=1* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 5* >* librbd::io::ImageRequestWQ: 0x7f88e8001570 unblock_writes: 0x5569b5e1ffd0,* >* num=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 notify released lock* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 current lock owner: [0,0]* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 handle_shutdown_post_release: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 wait_for_tracked_ops: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ManagedLock:* >* 0x7f88c4011bb8 complete_shutdown: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_shut_down_exclusive_lock: r=0* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_unregister_image_watcher* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::ImageWatcher:* >* 0x7f88c400dfe0 unregistering image watcher* >* >> > 2020-05-07T13:02:28.702+0300 7f88d4ff9700 10 librbd::Watcher:* >* 0x7f88c400dfe0 unregister_watch:* >* >> > 2020-05-07T13:02:28.702+0300 7f88d57fa700 5 librbd::Watcher:* >* 0x7f88c400dfe0 notifications_blocked: blocked=1* >* >> > 2020-05-07T13:02:28.706+0300 7f88ceffd700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_unregister_image_watcher: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88ceffd700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_flush_readahead* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_flush_readahead: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_shut_down_object_dispatcher* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::io::ObjectDispatcher: 0x5569b5dab700 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::io::ObjectDispatch: 0x5569b5ee8360 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::io::SimpleSchedulerObjectDispatch: 0x7f88c4013ce0 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 5* >* librbd::cache::WriteAroundObjectDispatch: 0x7f88c8003780 shut_down:* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0* >* handle_shut_down_object_dispatcher: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 send_flush_op_work_queue* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_flush_op_work_queue: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10* >* librbd::image::CloseRequest: 0x7f88c8175fd0 handle_flush_image_watcher: r=0* >* >> > 2020-05-07T13:02:28.706+0300 7f88d4ff9700 10 librbd::ImageState:* >* 0x5569b5da9bb0 0x5569b5da9bb0 handle_close: r=0* >* >> >* >* >> > On Fri, May 8, 2020 at 12:40 AM Brad Hubbard <bhubbard@xxxxxxxxxx>* >* wrote:* >* >> >>* >* >> >> On Fri, May 8, 2020 at 3:42 AM Erwin Lubbers <erwin@xxxxxxxxxxx>* >* wrote:* >* >> >> >* >* >> >> > Hi,* >* >> >> >* >* >> >> > Did anyone found a way to resolve the problem? I'm seeing the same* >* on a clean Octopus Ceph installation on Ubuntu 18 with an Octopus compiled* >* KVM server running on CentOS 7.8. The KVM machine shows:* >* >> >> >* >* >> >> > [ 7682.233684] fn-radosclient[6060]: segfault at 2b19 ip* >* 00007f8165cc0a50 sp 00007f81397f6490 error 4 in* >* librbd.so.1.12.0[7f8165ab4000+537000]* >* >> >>* >* >> >> Are you able to either capture a backtrace from a coredump or set up* >* >> >> logging and hopefully capture a backtrace that way?* >* >> >>* >* >> >> >* >* >> >> > Ceph is healthy and stable for a few weeks and I did not get these* >* messages while running on KVM compiled with Luminous libraries.* >* >> >> >* >* >> >> > Regards,* >* >> >> > Erwin* >* >> >> > _______________________________________________* >* >> >> > ceph-users mailing list -- ceph-users@xxxxxxx* >* >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx* >* >> >> >* >* >> >>* >* >> >>* >* >> >> --* >* >> >> Cheers,* >* >> >> Brad* >* >> >> _______________________________________________* >* >> >> ceph-users mailing list -- ceph-users@xxxxxxx* >* >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx* >* >>* >* >>* >* >>* >* >> --* >* >> Cheers,* >* >> Brad* >* >>* > > >* --* >* Cheers,* >* Brad*

4 4

ScholarOn
by glenda.davisn＠gmail.com 24 Sep '20

24 Sep '20

Students are welcome to utilize our <a href=" https://scholaron.com/subjects/human-resource-management ">Human Resource Management Answers</a> available on <a href=" https://scholaron.com/ ">Scholar On</a> with 100% quality and satisfaction guarantee. We aim at making all your educational endeavors fruitful with the help of multiple learning resources prepared with up-to-date academic needs of highered students.

1 0

Solve the issue of time scarcity via assignment help in Kuwait
by kevin wick 24 Sep '20

24 Sep '20

Are you facing a scarcity of timings or lack of time while composing assignments for Kuwait Universities? Are you not sure while taking the help of anybody to compose your assignments? For every student, assignments sound necessary but time-consuming tasks. You have to manage your time for your projects to score high marks as you can’t ignore your assignments during your study tenure. In this situation, if you can’t manage your time and require reliable assistance for your important task, place your order for assignment help even in Kuwait. Two important things that you need to keep in mind while working on your assignments are time management and quality content. Most students fail to submit their assignments on time because they could not manage their time and can’t collect relevant information for drafting their academic papers. However, don’t burst out and suffer your marks because of any reason. Instead of it, connect with assignment writing service provider and get your work done on time if you have less time to write your assignment. Many students achieve high marks on their projects due to the assistance of experts and professional writers. When you quote order for assignment writing help, you will provide enough time to engage yourself in some other academic tasks. If you don’t have time for writing your assignments and have no idea how to collect particulars for your work, transfer your project to professionals. Experts know how to arrange relevant information for composing the effective academic papers so you will not lose your marks. Expert’s knowledge and experience will allow you to connect with the right source of information and help you to score high marks. So, if you have issues in writing your academic papers, don’t forget to check out the services of online academic writing. https://www.greatassignmenthelp.com/kw/

3 2

ScholarOn
by glenda.davisn＠gmail.com 24 Sep '20

24 Sep '20

Students are welcome to utilize our <a href="https://scholaron.com/subjects/human-resource-management">Human Resource Management Answers</a> available on <a href="https://scholaron.com/">Scholar On</a> with 100% quality and satisfaction guarantee. We aim at making all your educational endeavors fruitful with the help of multiple learning resources prepared with up-to-date academic needs of highered students.

1 0

Module 'cephadm' has failed: auth get failed: failed to find osd.6 in keyring retval: -2
by bwolas＠gmail.com 24 Sep '20

24 Sep '20

Hello, I got this error after I tried upgrade from ceph:v15.2.4 to ceph:v15.2.5 with dead osd.6 cluster: id: 4e01640b-951b-4f75-8dca-0bad4faf1b11 health: HEALTH_ERR Module 'cephadm' has failed: auth get failed: failed to find osd.6 in keyring retval: -2 In short I added osd.6 on centos, but due network problems osd.6 failed to start, at a time I didint understand why and there was similar looking problem fixed in ceph:v15.2.5. So I tried to upgrade with cephadm, upgrade got stuck with this error, I canceled upgrade. Now osd.6 is fixed and working, but I still have this error. "ceph auth ls" shows osd.6 like every other osd it is working fine. I tried to upgrade again but this error keeps appearing in ceph -w and upgrade gets stuck again(everything is upgraded to ceph:v15.2.5 anyway) osd.6 is on separate host which has no other daemons.

1 1

OSD v15.2.5 daemon not starting on Centos7
by Christoph Ackermann 23 Sep '20

23 Sep '20

Hello, does anyone tried to update 15.2.4 on Centos7 to 15.2.5? I did a full: yum -y update on my first OSD node and after this no OSD on this want to start anymore. No log will be written, so osd process stops immediately i think. Starting osd daemon in forground shows that no tmpfs will be mountet under /var/lib/ceph/osd/ No kyering will be found! [root@ceph1n011 ~]# ceph-osd --id 10 2020-09-23T15:20:47.380+0200 7f1e0a018bc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2020-09-23T15:20:47.380+0200 7f1e0a018bc0 -1 AuthRegistry(0x560ee2ab8938) no keyring found at /var/lib/ceph/osd/ceph-10/keyring, disabling cephx 2020-09-23T15:20:47.382+0200 7f1e0a018bc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2020-09-23T15:20:47.382+0200 7f1e0a018bc0 -1 AuthRegistry(0x7ffe06f76400) no keyring found at /var/lib/ceph/osd/ceph-10/keyring, disabling cephx failed to fetch mon config (--no-mon-config to skip) I checked all systemd realated files and found no differences so far. This is quite strange and i look for some time to find the reason ..!? Other OSD on other (untouched) node shows this on startup.... [root@ceph1n012 ~]# ceph-osd --id 24 2020-09-23T15:15:23.514+0200 7f55ae95abc0 -1 Falling back to public interface 2020-09-23T15:15:43.143+0200 7f55ae95abc0 -1 osd.24 2154072 log_to_monitors {default=true} [root@ceph1n012 ~]# After some seconds osd process fork itself to background and works fine.... Does anybody have a clever hint for me? Best regards, Christoph -- Christoph Ackermann | System Engineer INFOSERVE GmbH | Am Felsbrunnen 15 | D-66119 Saarbrücken Fon +49 (0)681 88008-59 | Fax +49 (0)681 88008-33 | mailto:C.Ackermann@infoserve.de | https://www.infoserve.de INFOSERVE Datenschutzhinweise: https://infoserve.de/datenschutz Handelsregister: Amtsgericht Saarbrücken, HRB 11001 | Erfüllungsort: Saarbrücken Geschäftsführer: Dr. Stefan Leinenbach | Ust-IdNr.: DE168970599

1 1

Why seek help from an estate agent?
by devidpaul41＠gmail.com 23 Sep '20

23 Sep '20

Working with estate agents might not be the first choice for many of us. But if you are trying to find someone who can help you with your house for sale in Chichester or finding a renter, the estate agent can help you to meet potential investors and buyers. Estate agents in Chichester always ensure you get the best service. Opposed to the seemingly endless negotiations and contract signings for purchasing, estate agent make the process faster and do all authentication before bringing any offer in front of you. Estate agents in Chichester always act informative, helpful and responsive and look at tenants as a potential buyer. However, if you are looking for houses in Chichester to rent, ensure to check with the legal property owner and be permitted to let out the property. https://www.todanstee.com/cgi-bin/properties-for-sale-rent/rental-search.pl

1 0

Low level bluestore usage
by Ivan Kurnosov 23 Sep '20

23 Sep '20

Hi, this morning I woke up to a degraded test ceph cluster (managed by rook, but it does not really change anything for the question I'm about to ask). After checking logs I have found that bluestore on one of the OSDs run out of space. Some cluster details: ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) it runs on 3 little OSDs 10Gb each `ceph osd df` returned RAW USE of about 4.5GB on every node, happily reporting about 5.5GB of AVAIL. Yet: debug -9> 2020-09-22T20:23:15.421+0000 7f29e9798f40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1600806195424423, "job": 1, "event": "recovery_started", "log_files": [347, 350]} debug -8> 2020-09-22T20:23:15.421+0000 7f29e9798f40 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #347 mode 0 debug -7> 2020-09-22T20:23:16.465+0000 7f29e9798f40 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #350 mode 0 debug -6> 2020-09-22T20:23:18.689+0000 7f29e9798f40 1 bluefs _allocate failed to allocate 0x17a2360 on bdev 1, free 0x390000; fallback to bdev 2 debug -5> 2020-09-22T20:23:18.689+0000 7f29e9798f40 1 bluefs _allocate unable to allocate 0x17a2360 on bdev 2, free 0xffffffffffffffff; fallback to slow device expander debug -4> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluestore(/var/lib/ceph/osd/ceph-0) allocate_bluefs_freespace failed to allocate on 0x39a20000 min_size 0x17b0000 > allocated total 0x6250000 bluefs_shared_alloc_size 0x10000 allocated 0x0 available 0x 12ee32000 debug -3> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluefs _allocate failed to expand slow device to fit +0x17a2360 debug -2> 2020-09-22T20:23:18.689+0000 7f29e9798f40 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x17a2360 debug -1> 2020-09-22T20:23:18.693+0000 7f29e9798f40 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4/rpm/el8/BUILD/ceph-15.2.4/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f29e9798f40 time 2020-09-22T20:23:18.690014+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4/rpm/el8/BUILD/ceph-15.2.4/src/os/bluestore/BlueFS.cc: 2696: ceph_abort_msg("bluefs enospc") So, my question would be: how could I have prevented that? From monitoring I have (prometheus) - OSDs are healthy, have plenty of space, yet they are not. What command (and prometheus metric) would help me understand the actual real bluestore use? Or am I missing something? Oh, and I "fixed" the cluster by expanding the broken osd.0 with a larger 15GB volume. And 2 other OSDs still run on 10GB volumes. Thanks in advance for any thoughts. -- With best regards, Ivan Kurnosov

4 4

samba vfs_ceph: client_mds_namespace not working?
by Frank Schilder 23 Sep '20

23 Sep '20

Dear all, maybe someone has experienced this before. We are setting up a SAMBA gateway and would like to use the vfs_ceph module. In case of several file systems one needs to choose an mds namespace. There is an option in ceph.conf: client mds namespace = CEPH-FS-NAME Unfortunately, it seems not to work. I tried it in all possible versions, in [global] and [client], with and without "client" at the beginning, to no avail. I either get a time out or an error. I also found the libcephfs function ceph_select_filesystem(cmount, CEPH-FS-NAME) added it to vfs_ceph.c just before the ceph_mount with the same result, I get an error (operation not permitted). Does anyone know how to get this to work? And, yes, I tested an ordinary kernel fs mount with the credentials for the ceph client without problems. I can't access any documentation on the libcephfs api, I always get a page not found error. My last resort is now to ceph fs set-default CEPH-FS-NAME to the fs to be used and live with the implied restrictions and ugliness. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3

A disk move gone wrong & Luminous vs. Nautilus performance
by Nico Schottelius 23 Sep '20

23 Sep '20

Good morning, you might have seen my previous mails and I wanted to discuss some findings over the last day+night over what happened and why it happened here. As the system behaved inexplicitly for us, we are now looking for someone to analyse the root cause on consultancy basis - if you are interested & available, please drop me a PM. What happened ------------- Yesterday at about 1520 we moved 4 SSDs from server15 to server8. These SSDs are located in 3 SSD only pools. Soon after that the cluster begun to exhibit slow I/O on clients and saw many PGs in the peering state. Additionally we began to see slow ops on unrelated OSDs, i.e. osds that have a device class "hdd-big" set and are only selected by the "hdd" pool. Excerpt from ceph -s: 167 slow ops, oldest one blocked for 958 sec, daemons [osd.0,osd.11,osd.12,osd.14,osd.15,osd. 16,osd.19,osd.2,osd.22,osd.25]... have slow ops. All pgs that were hanging in peering state seemed to be related to pools from the moved SSDs. However the question remains why other osds began to be affected. Then we noticed that the newly started OSDs on server8 are consuming 100% cpu, the associated disks have a queue depth of 1 and the disks are 80-99% busy. This is a change that we noticed some weeks ago in the nautilus osd behaviour: while in luminous, osds would start right away, osds often take seconds to minutes to start in nautilus, accompanied with high cpu usage. Before yesterday however, the maximum startup time was in the range of minutes. When we noticed blocked/slow IOPS, we wanted to ensure that the clients are least possible affected. We tried to submit the two changes to all osds: ceph tell osd.* injectargs '--osd-recovery-max-active 1' ceph tell osd.* injectargs '--osd-max-backfills 1' However at this time, ceph tell would hang. We then switched to address each OSD individually using a for loop: for osd in $(ceph osd tree | grep osd. | awk '{ print $4 }'); do echo $osd; ( ceph tell $osd injectargs '--osd-max-backfills 1' &) ; done However this resulted in many hanging ceph tell processes and some of them began to report the following error: 2020-09-22 15:59:25.407 7fbc0359e700 0 --1- [2a0a:e5c0:2:1:20d:b9ff:fe48:3bd4]:0/1280010984 >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6856/4078 conn(0x7fbbec02edc0 0x7fbbec02f5d0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 connect got BADAUTHORIZER This message would be repeated many times a second per ceph tell process until we SIGINT killed it. After this failed we noticed that the remaining HDDs in server15 were at 100% i/o usage at about 0.6-2 MB/s reading rate. These low rates would very well explain the slow I/O on the clients - but it made us wonder why the rates were suddenly so low. We restarted one of the OSDs on this server, but the effect was that the osd would now use 100% cpu additionally to fully utilising the disk. We began to suspected a hardware problem and moved all HDDs to server8. Unfortunately, all newly HDD OSDs behave exactly the same: Very high cpu usage (60-100%), very high disk utilisation (about 90%+) and very low transfer rates (~1 MB/s on average). After a few hours (!) some of the OSDs normalised to 20-50MB/s read rates and after about 8h later all of them normalised. What is curious that during the slow phase of many hours, the average queue depth of the disk consistently stays at 1, while afterwards they usually average around 10-15. Given full utilisation, we suspect that the nautilus OSDs cause severly disk seeks. Our expectation (and so far behaviour with Luminous) was that if we have a single host failure, setting max backfills and max recovery to 1 and keeping the osd_recovery_op_priority low (usually at 2 in our clusters) has some, but more a minor impact on client I/O. This time, purely moving 4 osds with existing data made the cluster practically unusable, something we never experienced in a Luminous setup before. While I do not want to render out any potential mistakes or incorrect designs from our side, the effects seen are not expected from our side. While all of this happening we measured the network bandwidth on all nodes without congestion, monitored the monitors with some (spikes up to 50% from one core), but not high cpu usage. I was wondering whether anyone on the list has an insight on these questions: - Why were OSDs slow that are used in a different pool? - What changed in the OSDs from Luminous to Nautilus to make the startup phase slow? - What are nautilus OSDs doing at startup and why do they consume a lot of CPU? - What are the OSDs doing to severly impact the I/O? Is our seek theory correct? - Why does / can ceph tell hang in Nautilus? We never experienced this problem in Luminous. - Where does the BADAUTHORIZER message come from and what is the fix for it? - How can we debug / ensure that ceph tell does not hang? - Did code in the osds change that de-prioritizes command traffic? Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

1 0

Ceph RBD latency with synchronous writes?
by René Bartsch 23 Sep '20

23 Sep '20

Hi, we're considering running KVM virtual machine images on Ceph RBD block devices. How does Ceph RBD perform with the synchronous writes of databases (MariaDB)? Best regards, Renne

2 1

Vitastor, a fast Ceph-like block storage for VMs
by vitalif＠yourcmc.ru 23 Sep '20

23 Sep '20

Hi! After almost a year of development in my spare time I present my own software-defined block storage system: Vitastor - https://vitastor.io I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different coding schemes, rebalancing and so on. However it's much simpler and much faster. In a test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops saturation" parallel load benchmark it reached 895k read / 162k write iops, compared to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only using 4. Of course it's an early pre-release which means that, for example, it lacks snapshot support and other useful features. However the base is finished - it works and runs QEMU VMs. I like the design and I plan to develop it further. There are more details in the README file which currently opens from the domain https://vitastor.io Sorry if it was a bit off-topic, I just thought it could be interesting for you :) -- With best regards, Vitaliy Filippov

6 9

Update erasure code profile
by Thomas Svedberg 23 Sep '20

23 Sep '20

Hi, We are running a Nautilus cluster and have some old and new erasure code profiles. For example: # ceph osd erasure-code-profile get m_erasure crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get c_erasure crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 Here we want to add crush-device-class information to the second profile. Is the following correct? Safe? What will actually happen if we run this command? # ceph osd erasure-code-profile set c_erasure crush-device-class=hdd k=4 m=2 --force Thanks for any input! Regards, Thomas Svedberg

2 1

Documentation broken
by Frank Schilder 23 Sep '20

23 Sep '20

Hi all, during the migration of documentation, would it be possible to make the old documentation available somehow? A lot of pages are broken and I can't access the documentation for mimic at all any more. Is there an archive or something similar? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

switching to ceph-volume requires changing the default lvm.conf?
by Marc Roos 23 Sep '20

23 Sep '20

I was wondering if switching to ceph-volume requires me to change the default centos lvm.conf? Eg. The default has issue_discards = 0 Also I wonder if trimming is the default on lvm's on ssds? I read somewhere that the dmcrypt passthrough of trimming was still secure in combination with a btrfs filesystem.

1 0

Re: Troubleshooting stuck unclean PGs?
by Matt Larson 23 Sep '20

23 Sep '20

Hi Wout, None of the OSDs are greater than 20% full. However, only 1 PG is backfilling at a time, while the others are backfill_wait. I had recently added a large amount of data to the Ceph cluster, and this may have caused the # of PGs to increase causing the need to rebalance or move objects. It appears that I could increase the # of backfill operations that happen simultaneously by increasing `osd_max_backfills` and/or `osd_recovery_max_active`. It looks like I should maybe consider increasing the number of max backfills happening at a time because the overall io during the backfill is pretty small. Does this seem reasonable? If so, with Ceph Octopus/cephadm, how can adjust the parameters? Thanks, Matt On Mon, Sep 21, 2020 at 2:21 PM Wout van Heeswijk <wout(a)42on.com> wrote: > > Hi Matt, > > The mon data can grow during when PGs are stuck unclean. Don't restart the mons. > > You need to find out why your placement groups are "backfill_wait". Likely some of your OSDs are (near)full. > > If you have space elsewhere you can use the ceph balancer module or reweighting of OSDs to rebalance data. > > Scrubbing will continue once the PGs are "active+clean" > > Kind regards, > > Wout > 42on > > ________________________________________ > From: Matt Larson <larsonmattr(a)gmail.com> > Sent: Monday, September 21, 2020 6:22 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Troubleshooting stuck unclean PGs? > > Hi, > > Our Ceph cluster is reporting several PGs that have not been scrubbed > or deep scrubbed in time. It is over a week for these PGs to have been > scrubbed. When I checked the `ceph health detail`, there are 29 pgs > not deep-scrubbed in time and 22 pgs not scrubbed in time. I tried to > manually start a scrub on the PGs, but it appears that they are > actually in an unclean state that needs to be resolved first. > > This is a cluster running: > ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) > > Following the information at [Troubleshooting > PGs](https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-…, > I checked for PGs that are stuck stale | inactive | unclean. There > were no PGs that are stale or inactive, but there are several that are > stuck unclean: > > ``` > PG_STAT STATE UP > UP_PRIMARY ACTING ACTING_PRIMARY > 8.3c active+remapped+backfill_wait > [124,41,108,8,87,16,79,157,49] 124 > [139,57,16,125,154,65,109,86,45] 139 > 8.3e active+remapped+backfill_wait > [108,2,58,146,130,29,37,66,118] 108 > [127,92,24,50,33,6,130,66,149] 127 > 8.3f active+remapped+backfill_wait > [19,34,86,132,59,78,153,99,6] 19 > [90,45,147,4,105,61,30,66,125] 90 > 8.40 active+remapped+backfill_wait > [19,131,80,76,42,101,61,3,144] 19 > [28,106,132,3,151,36,65,60,83] 28 > 8.3a active+remapped+backfilling > [32,72,151,30,103,131,62,84,120] 32 > [91,60,7,133,101,117,78,20,158] 91 > 8.7e active+remapped+backfill_wait > [108,2,58,146,130,29,37,66,118] 108 > [127,92,24,50,33,6,130,66,149] 127 > 8.3b active+remapped+backfill_wait > [34,113,148,63,18,95,70,129,13] 34 > [66,17,132,90,14,52,101,47,115] 66 > 8.7f active+remapped+backfill_wait > [19,34,86,132,59,78,153,99,6] 19 > [90,45,147,4,105,61,30,66,125] 90 > 8.78 active+remapped+backfill_wait > [96,113,159,63,29,133,73,8,89] 96 > [138,121,15,103,55,41,146,69,18] 138 > 8.7d active+remapped+backfilling > [0,90,60,124,159,19,71,101,135] 0 > [150,72,124,129,63,10,94,29,41] 150 > 8.7c active+remapped+backfill_wait > [124,41,108,8,87,16,79,157,49] 124 > [139,57,16,125,154,65,109,86,45] 139 > 8.79 active+remapped+backfill_wait > [59,15,41,82,131,20,73,156,113] 59 > [13,51,120,102,29,149,42,79,132] 13 > ``` > > If I query one of the PGs that is backfilling, 8.3a, it shows it's state as : > "recovery_state": [ > { > "name": "Started/Primary/Active", > "enter_time": "2020-09-19T20:45:44.027759+0000", > "might_have_unfound": [], > "recovery_progress": { > "backfill_targets": [ > "30(3)", > "32(0)", > "62(6)", > "72(1)", > "84(7)", > "103(4)", > "120(8)", > "131(5)", > "151(2)" > ], > > Q1: Is there anything that I should check/fix to enable the PGs to > resolve from the `unclean` state? > Q2: I have also seen that the podman containers on one of our OSD > servers are taking large amounts of disk space. Is there a way to > limit the growth of disk space for podman containers, when > administering a Ceph cluster using `cephadm` tools? At last check, a > server running 16 OSDs and 1 MON is using 39G of disk space for its > running containers. Can restarting containers help to start with a > fresh slate or reduce the disk use? > > Thanks, > Matt > > ------------------------ > > Matt Larson > Associate Scientist > Computer Scientist/System Administrator > UW-Madison Cryo-EM Research Center > 433 Babcock Drive, Madison, WI 53706 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Matt Larson, PhD Madison, WI 53705 U.S.A.

4 7

How HP Printer Support help for troubleshooting?
by mary smith 23 Sep '20

23 Sep '20

To troubleshoot with the hp printer issue, you are required to download the HP Print and Scan Doctor from its official website. After that, you have to run HPPdr.exe from the download location on your system. And then, hit on the Start link and select your printer accordingly. Now, click fix printing. If there is an issue, use HP Printer Support. https://www.amiytech.com/hp-printer-support/

1 0

ceph-volume lvm cannot zap???
by Marc Roos 23 Sep '20

23 Sep '20

[@~]# ceph-volume lvm zap /dev/sdi --> Zapping: /dev/sdi --> --destroy was not specified, but zapping a whole device will remove the partition table stderr: wipefs: error: /dev/sdi: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition stderr: wipefs: error: /dev/sdi: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition I can see where it is busy, at least not in lsof

3 3

[nautilus] ceph tell hanging
by Nico Schottelius 23 Sep '20

23 Sep '20

Hello, recently we wanted to re-adjust rebalancing speed in one cluster with ceph tell osd.* injectargs '--osd-max-backfills 4' ceph tell osd.* injectargs '--osd-recovery-max-active 4' The first osds responded and after about 6-7 osds ceph tell stopped progressing, just after it encountered a dead osd (osd.10). We have since then removed osd.10 and all osds in the cluster are up. However as soon as we issue either of the above tell commands, it just hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in "Activating" and "Peering" states. It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few minutes later the pgs are peered/active. We can reproduce this problem also with very busy osds, which have been moved to another host - they also do not react to the ceph tell commands. We are mostly on 14.2.9, besides the rgw: [16:44:47] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } Did anyone see this before and/or do you have a hint on how to debug ceph tell as it is not a daemon on its own? Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

2 6

Unknown PGs after osd move
by Nico Schottelius 23 Sep '20

23 Sep '20

Hello, after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs: cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_WARN noscrub flag(s) set 2 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 241 pgs inactive 1532 slow requests are blocked > 32 sec 789 slow ops, oldest one blocked for 1949 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops. services: mon: 3 daemons, quorum black1,black2,black3 (age 97m) mgr: black2(active, since 96m), standbys: black1, black3 osd: 85 osds: 85 up, 82 in; 118 remapped pgs flags noscrub rgw: 1 daemon active (admin) data: pools: 12 pools, 3000 pgs objects: 33.96M objects, 129 TiB usage: 388 TiB used, 159 TiB / 548 TiB avail pgs: 8.033% pgs unknown 409151/101874117 objects misplaced (0.402%) 2634 active+clean 241 unknown 107 active+remapped+backfill_wait 11 active+remapped+backfilling 7 active+clean+scrubbing+deep io: client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr recovery: 67 MiB/s, 17 objects/s This used to be around 700+ unknown, however these 241 are stuck in this state for more than 1h. Below is a sample of pgs from "ceph pg dump all | grep unknown" 2.7f7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c2 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7ab 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.78b 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.788 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.76e 0 Using ceph pg 2.7f7 query hangs. We checked and one server did have an incorrect MTU setting (9204 instead of the correct 9000), but that was fixed some hours ago. Does anyone have a hint on how to find those unknown osds? Version wise this is 14.2.9: [20:42:20] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } From ceph health detail: [20:42:58] black2.place6:~# ceph health detail HEALTH_WARN noscrub flag(s) set; 2 nearfull osd(s); 1 pool(s) nearfull; Reduced data availability: 241 pgs inactive; 1575 slow requests are blocked > 32 sec; 751 slow ops, oldest one blocked for 1986 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.31,osd.33,osd.35]... have slow ops. OSDMAP_FLAGS noscrub flag(s) set OSD_NEARFULL 2 nearfull osd(s) osd.36 is near full osd.54 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'ssd' is nearfull PG_AVAILABILITY Reduced data availability: 241 pgs inactive pg 2.82 is stuck inactive for 6027.042489, current state unknown, last acting [] pg 2.88 is stuck inactive for 6027.042489, current state unknown, last acting [] ... pg 19.6e is stuck inactive for 6027.042489, current state unknown, last acting [] pg 20.69 is stuck inactive for 6027.042489, current state unknown, last acting [] As can be seen, multiple pools are affected even though most missing pgs are from pool 2. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

3 10

Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs
by Marc Roos 23 Sep '20

23 Sep '20

At least ceph thought you the essence of doing first proper testing ;) Because if you test your use case you either get a positive or negative result and not a problem. However I do have to admit that ceph could be more transparent with publishing testing and performance results. I have already discussed this with them on such a ceph day. It does not make sense to have to do everything yourself eg the luks overhead and putting the db/wal on ssd, rbd performance on hdds etc. Those can quickly show if ceph can be a candidate or not. -----Original Message----- From: Kevin Myers [mailto:response@ifastnet.com] Cc: Janne Johansson; Marc Roos; ceph-devel; ceph-users Subject: Re: [ceph-users] Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs Tbh ceph caused us more problems than it tried to fix ymmv good luck > On 22 Sep 2020, at 13:04, tri(a)postix.net wrote: > > The key is stored in the ceph cluster config db. It can be retrieved > by > > KEY=`/usr/bin/ceph --cluster ceph --name > client.osd-lockbox.${OSD_FSID} --keyring $OSD_PATH/lockbox.keyring > config-key get dm-crypt/osd/$OSD_FSID/luks` > > September 22, 2020 2:25 AM, "Janne Johansson" <icepic.dz(a)gmail.com> wrote: > >> Den mån 21 sep. 2020 kl 16:15 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: >> >>> When I create a new encrypted osd with ceph volume[1] >>> >>> Q4: Where is this luks passphrase stored? >> >> I think the OSD asks the mon for it after auth:ing, so "in the mon DBs" >> somewhere. >> >> -- >> May the most significant bit of your life be positive. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Ceph MDS stays in "up:replay" for hours. MDS failover takes 10-15 hours.
by heilig.oleg＠gmail.com 22 Sep '20

22 Sep '20

Hi there, We have 9 nodes Ceph cluster. Ceph version is 15.2.5. The cluster has 175 OSD (HDD) + 3 NVMe for cache tier for "cephfs_data" pool. CephFS pools info: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 350 TiB 179.53M 350 TiB 66.93 87 TiB cephfs_metadata 3 3.1 TiB 17.69M 3.1 TiB 1.77 87 TiB We use multiple active MDS instances: 3 "active" and 3 "standby". Each MDS server has 128GB RAM, "mds cache memory limit" = 64GB. Failover to a standby MDS instance takes 10-15 hours! CephFS is unreachable for the clients all this time. The MDS instance just stays in "up:replay" state for all this time. It looks like MDS demon checking all of the folders: 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EOpen.replay 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay 3 dirlumps by unknown.0 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay dir 0x300000041c5 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x300000041c5 /repository/files/14/ [2,head] auth v=2070324 cv=0/0 state=1610612737|complete f(v0 m2020-09-10T13:05:29.297254-0700 515=0+515) n(v46584 rc2020-09-21T20:38:49.071043-0700 b3937793650802 1056114=601470+454644) hs=515+0,ss=0+0 dirty=75 | child=1 subtree=0 dirty=1 0x55d4c9359b80] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay for [2,head] had [dentry #0x1/repository/files/14/14119 [2,head] auth (dversion lock) v=2049516 ino=0x30000812e2f state=1073741824 | inodepin=1 0x55db2463a1c0] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay for [2,head] had [inode 0x30000812e2f [...2,head] /repository/files/14/14119/ auth fragtree_t(*^3) v2049516 f(v0 m2020-09-18T10:17:53.379121-0700 13498=0+13498) n(v6535 rc2020-09-19T05:52:25.035403-0700 b272027384385 112669=81992+30677) (iversion lock) | dirfrag=8 0x55db24643000] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay dir 0x30000812e2f.000* 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x30000812e2f.000* /repository/files/14/14119/ [2,head] auth v=77082 cv=0/0 state=1073741824 f(v0 m2020-09-18T10:17:53.371122-0700 1636=0+1636) n(v6535 rc2020-09-19T05:51:18.063949-0700 b33321023818 13707=9986+3721) hs=885+0,ss=0+0 | child=1 0x55db845bf080] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/repository/files/14/14119/39823 [2,head] auth NULL (dversion lock) v=0 ino=(nil) state=1073741888|bottomlru 0x55d82061a900] We tried standby-replay and it helps but doesn't eliminate the root cause. We have millions folders with millions of small files. When the folders/subfolders scan is done, CephFS is active again. I believe 10 hours downtime is unexpected behaviour. Is there any way to force MDS to change status to active and run all of the required directory checks in the background? How can I localise the root cause?

2 1

one-liner getting block device from mounted osd
by Marc Roos 22 Sep '20

22 Sep '20

I have a optimize script that I run after the reboot of a ceph node. It sets among other things /sys/block/sdg/queue/read_ahead_kb and /sys/block/sdg/queue/nr_requests of block devices being used for osd's. Normally I am using the mount command to discover these but with the tmpfs and ceph-volume this does not work. Is anyone able to share a simple onliner getting from osd's the used block devices? Eg from both types at the same time. Or do I really need to traverse this block -> /dev/mapper/... ?

1 1

Re: rgw.none vs quota
by Jean-Sebastien Landry 22 Sep '20

22 Sep '20

Manuel & Konstantin, thank you to confirm this. I should upgrade to Nautilus in the next few weeks. I just live with it for now. Thanks!

1 0

Slow cluster and incorrect peers
by Nico Schottelius 22 Sep '20

22 Sep '20

Hello again, following up on the previous mail, one cluster gets rather slow at the moment and we have spotted something "funny": When checking ceph pg dump we see some osds have HB peers with osds that they should not have any pg in common with. When restarting one of the effected osds, we get the following message: mon_cmd_maybe_osd_create fail: 'osd.12 has already bound to class 'xxx-ssd', can not reset class to 'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy When checking the output of ceph osd tree, it seems to be in the correct class: 12 xxx-ssd 0.21767 osd.12 up 1.00000 1.00000 Is it possible that the osd has "multiple" classes / that the cluster remebers a class that was set to osd.12 when it used to be an HDD? The output of ceph pg dump includes at the bottom this OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 12 150 GiB 72 GiB 151 GiB 223 GiB [3,11,13,25,36,43,54,64,71,82] 128 35 which is wrong, because osd.12 should only peer with osd.3 and osd.25, which are the only ones in the same pool that has the replicated rule set to match on xxx-ssd. And the obvious question: how do we fix this? At the moment we see around 75 pgs in peering and 39 activating, most of them which are in a pool with slower SSDs, but it seems that these peerings affect another pool that should have faster SSDs. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

1 0

Mount CEPH-FS on multiple hosts with concurrent access to the same data objects?
by René Bartsch 22 Sep '20

22 Sep '20

I'm new on the list, so a "Hello" to all! :-) We're planning a Proxmox-Cluster. The data-center operator advised to use a virtual machine with NFS on top of a single CEPH-FS instance to mount the shared CEPH-FS storage on multiple hosts/VMs. As this NFS/CEPH-FS-VM could be a bottle-neck I was wondering if CEPH- FS is capable to manage concurrent access and locking itself. Is it possible to mount CEPH-FS instances on multiple hosts (e.g. /srv) all accessing the same data objects without data-loss or dead-locks by concurrent access? Will this perform better than a single NFS/CEPH-FS instance (VM)? Thanx for any hint Renne

5 7

RBD-Mirror: snapshots automatically created?
by Eugen Block 22 Sep '20

22 Sep '20

Hi *, I encountered a rather strange (or at least unexpected) behaviour of the rbd-mirror. Maybe I don't fully understand the feature so please correct me if my assumptions are wrong. My two (virtual one-node) clusters are still on ceph version 15.2.4-864-g0f510cb110 and the following happens if I'm trying to configure snapshot-based rbd-mirroring: - I create a one-way replication from site A to site B - Mirror mode is image (for snapshot-based mirroring) - Import image on site A (no journal-feature enabled to make sure) - Image is replicated as soon as I enable the mirror mode on that image with: > rbd mirror image enable pool/image3 snapshot - rbd info shows "snapshot_count: 1" although I haven't created a snapshot of this image yet, and there's also no schedule: > siteA:~ # rbd mirror snapshot schedule ls > siteA:~ # This is the third image in this test, I only created one snapshot for image1, not for the other images. Is this expected? From the docs I assumed I have to either create snapshots manually or configure the rbd mirror snapshot schedule. Could anyone please clarify? Thank you! Eugen

2 3

Re: Mount CEPH-FS on multiple hosts with concurrent access to the same data objects?
by Stefan Kooman 22 Sep '20

22 Sep '20

On 2020-09-21 21:12, Wout van Heeswijk wrote: > Hi Rene, > > Yes, cephfs is a good filesystem for concurrent writing. When using CephFS with ganesha you can even scale out. > > It will perform better but why don't you mount CephFS inside the VM? ^^ This. But it depends on the VMs you are going to use as clients. Do you trust those clients enough that they are allowed to be part of your cluster. Clients are really part of the cluster, at least that is how I see it. If possible, you want to use modern (5.7, 5.8) linux kernels for cephfs (rm operations: is slower on 4.15/5.3/5.4 for files created with 5.3/5.4 kernel). We have issues (sometimes) with older kernel clients (Ubuntu xenial, 4.15 kernel) and "MDS messages client failed to rdlock") but we don't have 100% proof yet it is because of this kernel version. They generally fix themselves though, so not a big issue. Gr. Stefan

2 1

ceph docs redirect not good
by Marc Roos 22 Sep '20

22 Sep '20

https://docs.ceph.com/docs/mimic/man/8/ceph-volume-systemd/

1 1

Setting up a small experimental CEPH network
by Philip Rhoades 22 Sep '20

22 Sep '20

People, I am interested in experimenting with CEPH on say 4 or 8 small form factor computers (SBCs?) - any suggestions about how to get started? I haven't bought anything yet - I have some working Fedora Workstations and Servers and a laptop but I don't want to experiment on them . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: phil(a)pricom.com.au

7 9

Troubleshooting stuck unclean PGs?
by Matt Larson 22 Sep '20

22 Sep '20

Hi, Our Ceph cluster is reporting several PGs that have not been scrubbed or deep scrubbed in time. It is over a week for these PGs to have been scrubbed. When I checked the `ceph health detail`, there are 29 pgs not deep-scrubbed in time and 22 pgs not scrubbed in time. I tried to manually start a scrub on the PGs, but it appears that they are actually in an unclean state that needs to be resolved first. This is a cluster running: ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) Following the information at [Troubleshooting PGs](https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-…, I checked for PGs that are stuck stale | inactive | unclean. There were no PGs that are stale or inactive, but there are several that are stuck unclean: ``` PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 8.3c active+remapped+backfill_wait [124,41,108,8,87,16,79,157,49] 124 [139,57,16,125,154,65,109,86,45] 139 8.3e active+remapped+backfill_wait [108,2,58,146,130,29,37,66,118] 108 [127,92,24,50,33,6,130,66,149] 127 8.3f active+remapped+backfill_wait [19,34,86,132,59,78,153,99,6] 19 [90,45,147,4,105,61,30,66,125] 90 8.40 active+remapped+backfill_wait [19,131,80,76,42,101,61,3,144] 19 [28,106,132,3,151,36,65,60,83] 28 8.3a active+remapped+backfilling [32,72,151,30,103,131,62,84,120] 32 [91,60,7,133,101,117,78,20,158] 91 8.7e active+remapped+backfill_wait [108,2,58,146,130,29,37,66,118] 108 [127,92,24,50,33,6,130,66,149] 127 8.3b active+remapped+backfill_wait [34,113,148,63,18,95,70,129,13] 34 [66,17,132,90,14,52,101,47,115] 66 8.7f active+remapped+backfill_wait [19,34,86,132,59,78,153,99,6] 19 [90,45,147,4,105,61,30,66,125] 90 8.78 active+remapped+backfill_wait [96,113,159,63,29,133,73,8,89] 96 [138,121,15,103,55,41,146,69,18] 138 8.7d active+remapped+backfilling [0,90,60,124,159,19,71,101,135] 0 [150,72,124,129,63,10,94,29,41] 150 8.7c active+remapped+backfill_wait [124,41,108,8,87,16,79,157,49] 124 [139,57,16,125,154,65,109,86,45] 139 8.79 active+remapped+backfill_wait [59,15,41,82,131,20,73,156,113] 59 [13,51,120,102,29,149,42,79,132] 13 ``` If I query one of the PGs that is backfilling, 8.3a, it shows it's state as : "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2020-09-19T20:45:44.027759+0000", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [ "30(3)", "32(0)", "62(6)", "72(1)", "84(7)", "103(4)", "120(8)", "131(5)", "151(2)" ], Q1: Is there anything that I should check/fix to enable the PGs to resolve from the `unclean` state? Q2: I have also seen that the podman containers on one of our OSD servers are taking large amounts of disk space. Is there a way to limit the growth of disk space for podman containers, when administering a Ceph cluster using `cephadm` tools? At last check, a server running 16 OSDs and 1 MON is using 39G of disk space for its running containers. Can restarting containers help to start with a fresh slate or reduce the disk use? Thanks, Matt ------------------------ Matt Larson Associate Scientist Computer Scientist/System Administrator UW-Madison Cryo-EM Research Center 433 Babcock Drive, Madison, WI 53706

2 1

What is the advice, one disk per OSD, or multiple disks
by Kees Bakker 21 Sep '20

21 Sep '20

Hello, Being new to CEPH, I need some advice how to setup a cluster. Given a node that has multiple disks, should I create one OSD for all disks, or is it better to have one OSD per disk. -- Kees Bakker

3 2

Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs
by Marc Roos 21 Sep '20

21 Sep '20

When I create a new encrypted osd with ceph volume[1] I assume something like this is being done, please correct what is wrong. - it creates the pv on the block device - it creates the ceph vg on the block device - it creates the osd lv in the vg - it uses cryptsetup to encrypt this lv (or is there some internal support for luks in lvm?) - it sets all the tags on the vg (shown by: lvs -o lv_tags vg) - it creates and enables ceph-volume@lvm-osdid-osdfsid - it creates and enables ceph-osd@osdid When a node is restarted, these lvm osds are started with - running ceph-volume@lvm-osdid-osdfsid (creating this tmpfs mount?) - running ceph-osd@osdid Q1: I had to create bootstrap-osd/ceph.keyring (ownership root.root). For what is that being used? Does it need to exist upon node restart? Q2: I had some issues with a node starting, solving this with adding a nofail to the fstab. How is this done with ceph-volume? Q3: Why these strange permissions on the mounted folder? drwxrwxrwt 2 ceph ceph 340 Sep 19 15:24 ceph-40 Q4: Where is this luks passphrase stored? Q5: Where does this tmpfs+content come from? How can I mount this myself from the command line? Q6: My lvm tags show ceph.crush_device_class=None, while ceph osd tree shows the correct class. Is this correct? Q7: I saw in my ceph-volume output sometimes 'disabling cephx', what does this mean? How can I verify this and fix it? Links to manuals are also welcome, these ceph-volume[2] are not to clear about this. [1] ceph-volume lvm create --data /dev/sdk --dmcrypt [2] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/

1 0

ceph 14.2.8 tracing ceph with blking compile error
by 陈晓波 21 Sep '20

21 Sep '20

I want to trace a request in ceph processing in 14.2.8 version. I open the blkin with do_cmake.sh -DWITH_BLKIN=ON, when compile there is a error in below: ../lib/libblkin.a(tp.c.o): undefined reference to symbol 'lttng_probe_register' //lib64/liblttng-ust.so.0: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status make[2]: *** [src/CMakeFiles/ceph-osd.dir/build.make:130: bin/ceph-osd] Error 1 make[1]: *** Waiting for unfinished jobs.......

1 0

Record recuperation mistake prompting Unlock Yahoo Account? Contact help group.
by mary smith 21 Sep '20

21 Sep '20

The record recuperation mistake can make a disappointment Unlock Yahoo Account. This issue can be settled by utilizing the assist that with canning be found in the tech consultancies. What's more, you can generally visit Youtube and take a gander at some tech vids that will help you in managing the issue. https://www.customercare-email.com/blog/unlock-my-yahoo-account/

1 0

Issue in the Epson interface? Find uphold from Epson Printer Support.
by mary smith 21 Sep '20

21 Sep '20

In the event that you face an interface issue in your gadget, by then you should reboot it. Regardless, on the off chance that the issue doesn't get unraveled, by then you can take help from the tech consultancies and utilize their investigating procedures. You can also contact the Epson Printer Support for finding support. https://www.epsonprintersupportpro.net/

1 0

Is ceph-mon disk write i/o normal at more than 1/2TB a day on an empty cluster?
by tri＠postix.net 20 Sep '20

20 Sep '20

Hi all, I'm in the process of testing out ceph on a small cluster. The cluster is virtually empty with no clients, just a few OSDs. I noticed extensive disk write I/O on some nodes and tracked them down to the ceph-mon daemon. A cursory log over 1 hour shows that each monitor generates about 22GB worth of disk write I/O or more than 1/2TB a day. The log was generated using iotop. It seems the 3 monitors in the cluster are doing the same thing. Is this normal or something I should be taking a closer look at? Cheers, --Tri Hoang

1 0

Process for adding a separate block.db to an osd
by tri＠postix.net 20 Sep '20

20 Sep '20

Hey all, I'm trying to figure out the appropriate process for adding a separate SSD block.db to an existing OSD. From what I gather the two steps are: 1. Use ceph-bluestore-tool bluefs-bdev-new-db to add the new db device 2. Migrate the data ceph-bluestore-tool bluefs-bdev-migrate I followed this and got both executed fine without any error. Yet when the OSD got started up, it keeps on using the integrated block.db instead of the new db. The block.db link to the new db device was deleted. Again, no error, just not using the new db Any suggestion? Thanks. --Tri Hoang root@elmo:/#CEPH_ARGS="--bluestore_block_db_size=26843545600 --bluestore_block_db_create=true" ceph-bluestore-tool --path /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2 bluefs-bdev-new-db --dev-target /dev/vg/sdc.db inferring bluefs devices from bluestore path DB device added /dev/dm-8 root@elmo:/# ceph-bluestore-tool --path /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2 --devs-source /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2/block --dev-target /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2/block.db bluefs-bdev-migrate inferring bluefs devices from bluestore path

2 2

Re: Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS
by Maged Mokhtar 20 Sep '20

20 Sep '20

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/dri… On 19/09/2020 10:31, huxiaoyu(a)horebdata.cn wrote: > Dear Maged, > > Thanks a lot for detailed explanantion on dm-writecache with Ceph. > > You mentioned REQ_FUA support patch for dm-writecache, does such a > patch not included into recent dm-writecache source code? I am using > 4.4 and 4.15/4.19 kernels, where do i get the mentioned patch? > > best regards, > > Samuel > > > > ------------------------------------------------------------------------ > huxiaoyu(a)horebdata.cn > > *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> > *Date:* 2020-09-18 18:20 > *To:* vitalif <mailto:vitalif@yourcmc.ru>; huxiaoyu > <mailto:huxiaoyu@horebdata.cn>; ceph-users <mailto:ceph-users@ceph.io> > *Subject:* Re: [ceph-users] Re: Benchmark WAL/DB on SSD and HDD > for RGW RBD CephFS > dm-writecache works using a high and low watermarks, set at 45 and > 50%. > All writes land in cache, once cache fills to the high watermark > backfilling to the slow device starts and stops when reaching the low > watermark. Backfilling uses b-tree with LRU blocks and tries merge > blocks to reduce hdd seeks, this is further helped by the io > scheduler > (cfq/deadline) ordering. > Each sync write op to the device requires 2 sync write ops, one > for data > and one for metadata, metadata is always in ram so there is no > additional metada read op (at the expense of using 2.5% of your cache > partition size in ram). So for pure sync writes (those with > REQ_FUA or > REQ_FLUSH which is used by Ceph) get half the SSD iops performance at > the device level. > Now the questions, what sustained performance would you get during > backfilling: it totally depends on whether your workload is > sequential > or random. For pure sequential workloads, all blocks are merged so > there > will not be a drop in input iops and backfilling occurs in small step > like intervals, but for such workloads you could get good performance > even without a cache. For purely random writes theoretically you > should > drop to the hdd random iops speed ( ie 80-150 iops ), but in our > testing > with fio pure random we would get 400-450 sustained iops, this is > probably related to the non-random-ness of fio rather than any magic. > For real life workloads that have a mix of both, this is where the > real > benefit of the cache will be felt, however it is not easy to simulate > such workloads, fio does offer a zipf/theta random distribution > control > but it was difficult for us to simulate real life workloads with > it, we > did some manual workloads such as installing and copying multiple vms > and we found the cache helped by 3-4 times the time to complete. > dm-writecache does serve reads if in cache, however the OSD cache > does > help for reads as well as any client read-ahead and in general writes > are the performance issue with hdd in Ceph. > For bcache, the only configuration we did was to enable write back > mode, > we did not set the block size to 4k. > If you want to try dm-writecache, use a recent 5.4+ kernel or a > kernel > with REQ_FUA support patch we did. You would need a recent lvm tools > package to support dm-writecache. We also limit the number of > backfill > blocks inflight to 100k blocks ie 400 MB. > /Maged > On 18/09/2020 13:38, vitalif(a)yourcmc.ru wrote: > >> we did test dm-cache, bcache and dm-writecache, we found the > later to be > >> much better. > > Did you set bcache block size to 4096 during your tests? Without > this setting it's slow because 99.9% SSDs don't handle 512 byte > overwrites well. Otherwise I don't think bcache should be worse > than dm-writecache. Also dm-writecache only caches writes, and > bcache also caches reads. And lvmcache is trash because it only > writes to SSD when the block is already on the SSD. > > > > Please post some details about the comparison if you have them :) >

1 0

Cephadm adoption not properly working
by Julian Fölsch 20 Sep '20

20 Sep '20

Good day everybody, We recently upgraded from Nautilus to Octopus and tried to adopt our cluster to cephadm following the guide in the documentation[0]. We only experienced some at first glance minor hickups where the cluster failed to refresh because it could not pull the container image[1]. This was fixed by setting `container_image` to the location where it could find the image in our network. But when we tried to run `ceph orch ps` it returned only a empty list. The one OSD we adopted after fixing the container_image variable was appearing fine in `ceph orch ps` then. However all Manager and Monitor daemons are now in a floating state where `cephadm ls` reports them as style "cephadm:v1"[2] but the orchestrator considers them stray[3]. Functionally however, those daemons just work fine and work together in the cluster. Can you help out how to get those daemons under the orchestrators control? Kind regards, Julian Fölsch [0] https://docs.ceph.com/en/octopus/cephadm/adoption/ [1] https://agdsn.me/~paktosan/ceph/adoption/image_failure.txt [2] https://agdsn.me/~paktosan/ceph/adoption/cephadm_ls.txt [3] https://agdsn.me/~paktosan/ceph/adoption/stray_daemons.txt -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

1 0

ceph-volume quite buggy compared to ceph-disk
by Marc Roos 19 Sep '20

19 Sep '20

[@]# ceph-volume lvm activate 36 82b94115-4dfb-4ed0-8801-def59a432b0a Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-36/lockbox.keyring --create-keyring --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --add-key AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ== stdout: creating /var/lib/ceph/osd/ceph-36/lockbox.keyring added entity client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a auth(key=AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==) Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/lockbox.keyring Running command: /usr/bin/ceph --cluster ceph --name client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --keyring /var/lib/ceph/osd/ceph-36/lockbox.keyring config-key get dm-crypt/osd/82b94115-4dfb-4ed0-8801-def59a432b0a/luks Running command: /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-9263e83b-7660-4f5b-843a-2111e882a17e/osd-block-82b94115-4dfb-4 ed0-8801-def59a432b0a I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb stderr: Device I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb already exists. Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb --path /var/lib/ceph/osd/ceph-36 --no-mon-config stderr: failed to read label for /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1 dmsetup ls lists this???? Where is an option to set the weight? As far as I can see you can only set this after peering started? How can I mount this tmpfs manually to inspect this? Maybe put in the manual[1]? [1] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/

1 0

Re: Ceph RDMA GID Selection Problem
by Lazuardi Nasution 19 Sep '20

19 Sep '20

Hi Samuel, I resent my reply since CC will not be accepted by this mailing list. I'm using Octopus. If it is the release problem, why it is successful with only ms_async_rdma_device_name and ms_async_rdma_gid_idx? Best regards. On Sat, Sep 19, 2020 at 3:33 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > Which Ceph version are you using? Just wondering Ceph RDMA support is > officially announced, or still on development? > > best regards, > > samuel > > ------------------------------ > huxiaoyu(a)horebdata.cn > > > *From:* Lazuardi Nasution <mrxlazuardin(a)gmail.com> > *Date:* 2020-09-18 19:21 > *To:* ceph-users <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Ceph RDMA GID Selection Problem > Hi, > > I have something weird about GID selection for Ceph with RDMA. When I do > configuration with ms_async_rdma_device_name and ms_async_rdma_gid_idx, > Ceph with RDMA running successfully. But, when I do configuration with > ms_async_rdma_device_name, ms_async_rdma_local_gid and > ms_async_rdma_roce_ver, Ceph with RDMA is not working, OSDs are down in > seconds after they up. The GID index which is used on the first attempt is > associated with the GID and RoCE version which are used on the second > attempt. Is this the string matter (maybe because GID is using colon > characters) or something else? Using the GID index sometimes gives me > problems due to it not persisting and changes happen every time I do > network reconfiguration (for example: adding/removing VLAN), or even > rebooting. > > Best regards, > >

1 0

Re: Ceph RDMA GID Selection Problem
by Lazuardi Nasution 19 Sep '20

19 Sep '20

Hi Samuel, I'm using Octopus. If it is the release problem, why it is successful with only ms_async_rdma_device_name and ms_async_rdma_gid_idx? Best regards, On Sat, Sep 19, 2020 at 3:33 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > Which Ceph version are you using? Just wondering Ceph RDMA support is > officially announced, or still on development? > > best regards, > > samuel > > ------------------------------ > huxiaoyu(a)horebdata.cn > > > *From:* Lazuardi Nasution <mrxlazuardin(a)gmail.com> > *Date:* 2020-09-18 19:21 > *To:* ceph-users <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Ceph RDMA GID Selection Problem > Hi, > > I have something weird about GID selection for Ceph with RDMA. When I do > configuration with ms_async_rdma_device_name and ms_async_rdma_gid_idx, > Ceph with RDMA running successfully. But, when I do configuration with > ms_async_rdma_device_name, ms_async_rdma_local_gid and > ms_async_rdma_roce_ver, Ceph with RDMA is not working, OSDs are down in > seconds after they up. The GID index which is used on the first attempt is > associated with the GID and RoCE version which are used on the second > attempt. Is this the string matter (maybe because GID is using colon > characters) or something else? Using the GID index sometimes gives me > problems due to it not persisting and changes happen every time I do > network reconfiguration (for example: adding/removing VLAN), or even > rebooting. > > Best regards, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >

1 0

Ceph RDMA GID Selection Problem
by Lazuardi Nasution 19 Sep '20

19 Sep '20

Hi, I have something weird about GID selection for Ceph with RDMA. When I do configuration with ms_async_rdma_device_name and ms_async_rdma_gid_idx, Ceph with RDMA running successfully. But, when I do configuration with ms_async_rdma_device_name, ms_async_rdma_local_gid and ms_async_rdma_roce_ver, Ceph with RDMA is not working, OSDs are down in seconds after they up. The GID index which is used on the first attempt is associated with the GID and RoCE version which are used on the second attempt. Is this the string matter (maybe because GID is using colon characters) or something else? Using the GID index sometimes gives me problems due to it not persisting and changes happen every time I do network reconfiguration (for example: adding/removing VLAN), or even rebooting. Best regards,

2 1

Re: Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS
by vitalif＠yourcmc.ru 19 Sep '20

19 Sep '20

> we did test dm-cache, bcache and dm-writecache, we found the later to be > much better. Did you set bcache block size to 4096 during your tests? Without this setting it's slow because 99.9% SSDs don't handle 512 byte overwrites well. Otherwise I don't think bcache should be worse than dm-writecache. Also dm-writecache only caches writes, and bcache also caches reads. And lvmcache is trash because it only writes to SSD when the block is already on the SSD. Please post some details about the comparison if you have them :)

3 2