October 2020 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

fixing future rctimes
by Dan van der Ster 23 Mar '21

23 Mar '21

Hi all, We have a few subdirs with an rctime in the future. # getfattr -n ceph.dir.rctime session # file: session ceph.dir.rctime="2576387188.090" I can't find any subdir or item in that directory with that rctime, so I presume that there was previously a file and that rctime cannot go backwards [1] Is there any way to fix these rctimes so they show the latest ctime of the subtree? Also -- are we still relying on the client clock to set the rctime / ctime of a file? Would it make sense to limit ctime/rctime for any update to the current time on the MDS ? Best Regards, Dan [1] https://github.com/ceph/ceph/pull/24023/commits/920ef964311a61fcc6c0d6671b7…

2 3

Remapped PGs
by David Orman 02 Mar '21

02 Mar '21

Hi, We see that we have 5 'remapped' PGs, but are unclear why/what to do about it. We shifted some target ratios for the autobalancer and it resulted in this state. When adjusting ratio, we noticed two OSDs go down, but we just restarted the container for those OSDs with podman, and they came back up. Here's status output: ################### root@ceph01:~# ceph status INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_OK services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w) mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl, ceph02.gcglcg, ceph04.smbdew, ceph05.yropto osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs data: pools: 3 pools, 1057 pgs objects: 18.00M objects, 69 TiB usage: 119 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 1056 active+clean 1 active+clean+scrubbing+deep io: client: 859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr root@ceph01:~# ################### When I look at ceph pg dump, I don't see any marked as remapped: ################### root@ceph01:~# ceph pg dump |grep remapped INFO:cephadm:Inferring fsid x INFO:cephadm:Inferring config x INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 dumped all root@ceph01:~# ################### Any idea what might be going on/how to recover? All OSDs are up. Health is 'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman 2.0.3.

2 4

Re: mds lost very frequently
by Stefan Kooman 24 Feb '21

24 Feb '21

Hi, After setting: ceph config set mds mds_recall_max_caps 10000 (5000 before change) and ceph config set mds mds_recall_max_decay_rate 1.0 (2.5 before change) And the: ceph tell 'mds.*' injectargs '--mds_recall_max_caps 10000' ceph tell 'mds.*' injectargs '--mds_recall_max_decay_rate 1.0' our up:active MDS stopped responding and the standby-replay stepped in ... and hit an assert (same as in this thread): 2020-02-06 16:42:16.712 7ff76a528700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-02-06 16:42:17.616 7ff76ff1b700 0 mds.beacon.mds2 MDS is no longer laggy 2020-02-06 16:42:20.348 7ff76d716700 -1 /build/ceph-13.2.8/src/mds/Locker.cc: In function 'void Locker::file_recover(ScatterLock*)' thread 7ff76d716700 time 2020-02-06 16:42:20.351124 /build/ceph-13.2.8/src/mds/Locker.cc: 5307: FAILED assert(lock->get_state() == LOCK_PRE_SCAN) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7ff7759939de] 2: (()+0x287b67) [0x7ff775993b67] 3: (()+0x28a9ea) [0x5585eb2b79ea] 4: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 5: (MDSRank::active_start()+0x135) [0x5585eb146be5] 6: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 7: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 8: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 9: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 10: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 12: (()+0x76db) [0x7ff7752846db] 13: (clone()+0x3f) [0x7ff77446a88f] 2020-02-06 16:42:20.348 7ff76d716700 -1 *** Caught signal (Aborted) ** in thread 7ff76d716700 thread_name:ms_dispatch ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0x12890) [0x7ff77528f890] 2: (gsignal()+0xc7) [0x7ff774387e97] 3: (abort()+0x141) [0x7ff774389801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7ff775993ae6] 5: (()+0x287b67) [0x7ff775993b67] 6: (()+0x28a9ea) [0x5585eb2b79ea] 7: (MDCache::start_files_to_recover()+0xbb) [0x5585eb1f897b] 8: (MDSRank::active_start()+0x135) [0x5585eb146be5] 9: (MDSRankDispatcher::handle_mds_map(MMDSMap*, MDSMap*)+0x4e5) [0x5585eb151ea5] 10: (MDSDaemon::handle_mds_map(MMDSMap*)+0xca8) [0x5585eb134608] 11: (MDSDaemon::handle_core_message(Message*)+0x6c) [0x5585eb138bbc] 12: (MDSDaemon::ms_dispatch(Message*)+0xbb) [0x5585eb13929b] 13: (DispatchQueue::entry()+0xb92) [0x7ff775a56e52] 14: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff775af3e2d] 15: (()+0x76db) [0x7ff7752846db] 16: (clone()+0x3f) [0x7ff77446a88f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Quoting Yan, Zheng (ukernel(a)gmail.com) > Please try below patch if you can compile ceph from source. If you > can't compile ceph or the issue still happens, please set debug_mds = > 10 for standby mds (change debug_mds to 0 after mds becomes active). > > Regards > Yan, Zheng > > diff --git a/src/mds/MDSRank.cc b/src/mds/MDSRank.cc > index 1e8b024b8a..d1150578f1 100644 > --- a/src/mds/MDSRank.cc > +++ b/src/mds/MDSRank.cc > @@ -1454,8 +1454,8 @@ void MDSRank::rejoin_done() > void MDSRank::clientreplay_start() > { > dout(1) << "clientreplay_start" << dendl; > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > queue_one_replay(); > } > > @@ -1487,8 +1487,8 @@ void MDSRank::active_start() > > mdcache->clean_open_file_lists(); > mdcache->export_remaining_imported_caps(); > - finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > mdcache->start_files_to_recover(); > + finish_contexts(g_ceph_context, waiting_for_replay); // kick waiters > > mdcache->reissue_all_caps(); > mdcache->activate_stray_manager(); AFAICT this patch has never been tested and never commited. Do you still think this might fix the issue? Any hints on how we might reproduce this issue: failing active mds and hitting this specific recovery scenario We will happily apply this patch and do testing to check if it really fixes the issue. Gr. Stefan P.s. For my understanding: the MDS should never stop responding by setting these parameters, right? -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

Default data pool in CEPH
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi, I have a CEPH 15.2.4 running in a docker. How to configure for use a specific data pool? i try put the follow line in the ceph.conf but the changes not working. . [client.myclient] rbd default data pool = Mydatapool I need it to configure for erasure pool with cloudstack Can help me? , where is the ceph conf we i need configure? Thanks. -- Untitled Document

4 6

Re: NFS Ganesha NFSv3
by Gabriel Medve 24 Feb '21

24 Feb '21

Hi Thanks for the reply. cephadm runs ceph containers automatically. How to set privileged mode in ceph container? -- > El 23/9/20 a las 13:24, Daniel Gryniewicz escribió: >> NFSv3 needs privileges to connect to the portmapper. Try running >> your docker container in privileged mode, and see if that helps. >> >> Daniel >> >> On 9/23/20 11:42 AM, Gabriel Medve wrote: >>> Hi, >>> >>> I have a CEPH 15.2.5 running in a docker , i configure nfs ganesha >>> with nfs version 3 but i can not mount it. >>> If configure ganesha with nfs version 4 i can mounted without >>> problems but i need the version 3 . >>> >>> The error is mount.nfs: Protocol not supported >>> >>> Can help me? >>> >>> Thanks. >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- > Untitled Document

2 1

Possible to disable check: x pool(s) have no replicas configured
by Marc Roos 23 Feb '21

23 Feb '21

Is it possible to disable checking on 'x pool(s) have no replicas configured', so I don't have this HEALTH_WARN constantly. Or is there some other disadvantage of keeping some empty 1x replication test pools?

2 1

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Bluestore migration: per-osd device copy
by Chris Dunlop 05 Jan '21

05 Jan '21

Hi, The docs have scant detail on doing a migration to bluestore using a per-osd device copy: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-o… This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a copy function (all the way from v9 to current). Has anyone actually tried doing this? Is there any further detail available on what is involved, e.g. a broad outline of the steps? Of course, detailed instructions would be even better, even if accompanied by "here be dragons!" warnings. Cheers, Chris

3 6

diskprediction_local fails with python3-sklearn 0.22.2
by Eric Dold 22 Dec '20

22 Dec '20

Hello the mgr module diskprediction_local fails under ubuntu 20.04 focal with python3-sklearn version 0.22.2 Ceph version is 15.2.3 when the module is enabled i get the following error: File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in serve self.predict_all_devices() File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in predict_all_devices result = self._predict_life_expentancy(devInfo['devid']) File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in _predict_life_expentancy predicted_result = obj_predictor.predict(predict_datas) File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457, in predict pred = clf.predict(ordered_data) File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in predict if self.break_ties and self.decision_function_shape == 'ovo': AttributeError: 'SVC' object has no attribute 'break_ties' Best Regards Eric

2 1

v15.2.4 Octopus released
by David Galloway 17 Dec '20

17 Dec '20

We're happy to announce the fourth bugfix release in the Octopus series. In addition to a security fix in RGW, this release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-4-octopus-released Notable Changes --------------- * CVE-2020-10753: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader (William Bowling, Adam Mohammed, Casey Bodley) * Cephadm: There were a lot of small usability improvements and bug fixes: * Grafana when deployed by Cephadm now binds to all network interfaces. * `cephadm check-host` now prints all detected problems at once. * Cephadm now calls `ceph dashboard set-grafana-api-ssl-verify false` when generating an SSL certificate for Grafana. * The Alertmanager is now correctly pointed to the Ceph Dashboard * `cephadm adopt` now supports adopting an Alertmanager * `ceph orch ps` now supports filtering by service name * `ceph orch host ls` now marks hosts as offline, if they are not accessible. * Cephadm can now deploy NFS Ganesha services. For example, to deploy NFS with a service id of mynfs, that will use the RADOS pool nfs-ganesha and namespace nfs-ns:: ceph orch apply nfs mynfs nfs-ganesha nfs-ns * Cephadm: `ceph orch ls --export` now returns all service specifications in yaml representation that is consumable by `ceph orch apply`. In addition, the commands `orch ps` and `orch ls` now support `--format yaml` and `--format json-pretty`. * Cephadm: `ceph orch apply osd` supports a `--preview` flag that prints a preview of the OSD specification before deploying OSDs. This makes it possible to verify that the specification is correct, before applying it. * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, and `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * RBD: The name of the rbd pool object that is used to store rbd trash purge schedule is changed from "rbd_trash_trash_purge_schedule" to "rbd_trash_purge_schedule". Users that have already started using `rbd trash purge schedule` functionality and have per pool or namespace schedules configured should copy "rbd_trash_trash_purge_schedule" object to "rbd_trash_purge_schedule" before the upgrade and remove "rbd_trash_purge_schedule" using the following commands in every RBD pool and namespace where a trash purge schedule was previously configured:: rados -p <pool-name> [-N namespace] cp rbd_trash_trash_purge_schedule rbd_trash_purge_schedule rados -p <pool-name> [-N namespace] rm rbd_trash_trash_purge_schedule or use any other convenient way to restore the schedule after the upgrade. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 7447c15c6ff58d7fce91843b705a268a1917325c -- David Galloway Systems Administrator, RDU Ceph Engineering IRC: dgalloway

4 5

multiple OSD crash, unfound objects
by Michael Thomas 15 Dec '20

15 Dec '20

Over the weekend I had multiple OSD servers in my Octopus cluster (15.2.4) crash and reboot at nearly the same time. The OSDs are part of an erasure coded pool. At the time the cluster had been busy with a long-running (~week) remapping of a large number of PGs after I incrementally added more OSDs to the cluster. After bringing all of the OSDs back up, I have 25 unfound objects and 75 degraded objects. There are other problems reported, but I'm primarily concerned with these unfound/degraded objects. The pool with the missing objects is a cephfs pool. The files stored in the pool are backed up on tape, so I can easily restore individual files as needed (though I would not want to restore the entire filesystem). I tried following the guide at https://docs.ceph.com/docs/octopus/rados/troubleshooting/troubleshooting-pg…. I found a number of OSDs that are still 'not queried'. Restarting a sampling of these OSDs changed the state from 'not queried' to 'already probed', but that did not recover any of the unfound or degraded objects. I have also tried 'ceph pg deep-scrub' on the affected PGs, but never saw them get scrubbed. I also tried doing a 'ceph pg force-recovery' on the affected PGs, but only one seems to have been tagged accordingly (see ceph -s output below). The guide also says "Sometimes it simply takes some time for the cluster to query possible locations." I'm not sure how long "some time" might take, but it hasn't changed after several hours. My questions are: * Is there a way to force the cluster to query the possible locations sooner? * Is it possible to identify the files in cephfs that are affected, so that I could delete only the affected files and restore them from backup tapes? --Mike ceph -s: cluster: id: 066f558c-6789-4a93-aaf1-5af1ba01a3ad health: HEALTH_ERR 1 clients failing to respond to capability release 1 MDSs report slow requests 25/78520351 objects unfound (0.000%) 2 nearfull osd(s) Reduced data availability: 1 pg inactive Possible data damage: 9 pgs recovery_unfound Degraded data redundancy: 75/626645098 objects degraded (0.000%), 9 pgs degraded 1013 pgs not deep-scrubbed in time 1013 pgs not scrubbed in time 2 pool(s) nearfull 1 daemons have recently crashed 4 slow ops, oldest one blocked for 77939 sec, daemons [osd.0,osd.41] have slow ops. services: mon: 4 daemons, quorum ceph1,ceph2,ceph3,ceph4 (age 9d) mgr: ceph3(active, since 11d), standbys: ceph2, ceph4, ceph1 mds: archive:1 {0=ceph4=up:active} 3 up:standby osd: 121 osds: 121 up (since 6m), 121 in (since 101m); 4 remapped pgs task status: scrub status: mds.ceph4: idle data: pools: 9 pools, 2433 pgs objects: 78.52M objects, 298 TiB usage: 412 TiB used, 545 TiB / 956 TiB avail pgs: 0.041% pgs unknown 75/626645098 objects degraded (0.000%) 135224/626645098 objects misplaced (0.022%) 25/78520351 objects unfound (0.000%) 2421 active+clean 5 active+recovery_unfound+degraded 3 active+recovery_unfound+degraded+remapped 2 active+clean+scrubbing+deep 1 unknown 1 active+forced_recovery+recovery_unfound+degraded progress: PG autoscaler decreasing pool 7 PGs from 1024 to 512 (5d) [............................]

4 34

Provide more documentation for MDS performance tuning on large file systems
by Janek Bevendorff 15 Dec '20

15 Dec '20

Hello, Over the last week I have tried optimising the performance of our MDS nodes for the large amount of files and concurrent clients we have. It turns out that despite various stability fixes in recent releases, the default configuration still doesn't appear to be optimal for keeping the cache size under control and avoid intermittent I/O blocks. Unfortunately, it is very hard to tweak the configuration to something that works, because the tuning parameters needed are largely undocumented or only described in very technical terms in the source code making them quite unapproachable for administrators not familiar with all the CephFS internals. I would therefore like to ask if it were possible to document the "advanced" MDS settings more clearly as to what they do and in what direction they have to be tuned for more or less aggressive cap recall, for instance (sometimes it is not clear if a threshold is a min or a max threshold). I am am in the very (un)fortunate situation to have folders with a several 100K direct sub folders or files (and one extreme case with almost 7 million dentries), which is a pretty good benchmark for measuring cap growth while performing operations on them. For the time being, I came up with this configuration, which seems to work for me, but is still far from optimal: mds basic mds_cache_memory_limit 10737418240 mds advanced mds_cache_trim_threshold 131072 mds advanced mds_max_caps_per_client 500000 mds advanced mds_recall_max_caps 17408 mds advanced mds_recall_max_decay_rate 2.000000 The parameters I am least sure about---because I understand the least how they actually work---are mds_cache_trim_threshold and mds_recall_max_decay_rate. Despite reading the description in src/common/options.cc, I understand only half of what they're doing and I am also not quite sure in which direction to tune them for optimal results. Another point where I am struggling is the correct configuration of mds_recall_max_caps. The default of 5K doesn't work too well for me, but values above 20K also don't seem to be a good choice. While high values result in fewer blocked ops and better performance without destabilising the MDS, they also lead to slow but unbounded cache growth, which seems counter-intuitive. 17K was the maximum I could go. Higher values work for most use cases, but when listing very large folders with millions of dentries, the MDS cache size slowly starts to exceed the limit after a few hours, since the MDSs are failing to keep clients below mds_max_caps_per_client despite not showing any "failing to respond to cache pressure" warnings. With the configuration above, I do not have cache size issues any more, but it comes at the cost of performance and slow/blocked ops. A few hints as to how I could optimise my settings for better client performance would be much appreciated and so would be additional documentation for all the "advanced" MDS settings. Thanks a lot Janek

3 13

rgw index shard much larger than others
by Dan van der Ster 08 Dec '20

08 Dec '20

Dear friends, Running 14.2.11, we have one particularly large bucket with a very strange distribution of objects among the shards. The bucket has 512 shards, and most shards have ~75k entries, but shard 0 has 1.75M entries: # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.0 | wc -l 1752085 # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.1 | wc -l 78388 # rados -p default.rgw.buckets.index listomapkeys .dir.61c59385-085d-4caa-9070-63a3868dccb6.272652427.1.2 | wc -l 78764 We had resharded this bucket (manually) from 32 up to 512 shards just before upgrading from 12.2.12 to 14.2.11 a couple weeks ago. Any idea why shard .0 is getting such an imbalance of entries? Should we manually reshard this bucket again? Thanks! Dan

3 6

block.db/block.wal device performance dropped after upgrade to 14.2.10
by Vladimir Prokofev 04 Dec '20

04 Dec '20

Good day, cephers! We've recently upgraded our cluster from 14.2.8 to 14.2.10 release, also performing full system packages upgrade(Ubuntu 18.04 LTS). After that performance significantly dropped, main reason beeing that journal SSDs are now have no merges, huge queues, and increased latency. There's a few screenshots in attachments. This is for an SSD journal that supports block.db/block.wal for 3 spinning OSDs, and it looks like this for all our SSD block.db/wal devices across all nodes. Any ideas what may cause that? Maybe I've missed something important in release notes?

6 20

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

How to create single OSD with SSD db device with cephadm
by 胡玮文 03 Dec '20

03 Dec '20

Hi all, I’m new to ceph. I recently deployed a ceph cluster with cephadm. Now I want to add a single new OSD daemon with a db device on SSD. But I can’t find any documentation about this. I have tried: 1. Using web dashboard. This requires at least one filter to proceed (type, vendor, model or size). But I just want to select the block device manually. 2. Using ‘ceph orch apply osd -i spec.yml’. This is also filter based. 3. Using ‘ceph orch daemon add osd host:device’. Seems I cannot specify my SSD db device in this way. 4. On the target host, run ‘cephadm shell’ then ceph-volume prepare and activate. But ceph-volume seems can’t create systemd service outside the container like ‘ceph orch’ does. 5. On the target host, run ‘cephadm ceph-volume’, but it requires a json config file, I can’t figure out what is that. Any help is appreciated. Thanks.

2 4

Advice on SSD choices for WAL/DB?
by Andrei Mikhailovsky 26 Nov '20

26 Nov '20

Hello, We are planning to perform a small upgrade to our cluster and slowly start adding 12TB SATA HDD drives. We need to accommodate for additional SSD WAL/DB requirements as well. Currently we are considering the following: HDD Drives - Seagate EXOS 12TB SSD Drives for WAL/DB - Intel D3 S4510 960GB or Intel D3 S4610 960GB Our cluster isn't hosting any IO intensive DBs nor IO hungry VMs such as Exchange, MSSQL, etc. From the documentation that I've read the recommended size for DB is between 1% and 4% of the size of the osd. Would 2% figure be sufficient enough (so around 240GB DB size for each 12TB osd?) Also, from your experience, which is a better model for the SSD DB/WAL? Would Intel S4510 be sufficient enough for our purpose or would the S4610 be a much better choice? Are there any other cost effective performance to consider instead of the above models? The same question to the HDD. Any other drives we should consider instead of the Seagate EXOS series? Thanks for you help and suggestions. Andrei

4 4

Problems with mon
by Mateusz Skała 21 Nov '20

21 Nov '20

Hello Community, I have problems with ceph-mons in docker. Docker pods are starting but I got a lot of messages "e6 handle_auth_request failed to assign global_id” in log. 2 mons are up but I can’t send any ceph commands. Regards Mateusz

2 4

Not all OSDs in rack marked as down when the rack fails
by Wido den Hollander 18 Nov '20

18 Nov '20

Hi, I'm investigating an issue where 4 to 5 OSDs in a rack aren't marked as down when the network is cut to that rack. Situation: - Nautilus cluster - 3 racks - 120 OSDs, 40 per rack We performed a test where we turned off the network Top-of-Rack for each rack. This worked as expected with two racks, but with the third something weird happened. From the 40 OSDs which were supposed to be marked as down only 36 were marked as down. In the end it took 15 minutes for all 40 OSDs to be marked as down. $ ceph config set mon mon_osd_reporter_subtree_level rack That setting is set to make sure that we only accept reports from other racks. What we saw in the logs for example: 2020-10-29T03:49:44.409-0400 7fbda185e700 10 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 osd.51 has 54 reporters, 239.856038 grace (20.000000 + 219.856 + 7.43801e-23), max_failed_since 2020-10-29T03:47:22.374857-0400 But osd.51 was still not marked as down after 54 reporters have reported that it is actually down. I checked, no ping or other traffic possible to osd.51. Host is unreachable. Another osd was marked as down, but it took a couple of minutes as well: 2020-10-29T03:50:54.455-0400 7fbda185e700 10 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 osd.37 has 48 reporters, 221.378970 grace (20.000000 + 201.379 + 6.34437e-23), max_failed_since 2020-10-29T03:47:12.761584-0400 2020-10-29T03:50:54.455-0400 7fbda185e700 1 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107102 we have enough reporters to mark osd.37 down In the end osd.51 was marked as down, but only after the MON decided to do so: 2020-10-29T03:53:44.631-0400 7fbda185e700 0 log_channel(cluster) log [INF] : osd.51 marked down after no beacon for 903.943390 seconds 2020-10-29T03:53:44.631-0400 7fbda185e700 -1 mon.CEPH2-MON1-206-U39(a)0(leader).osd e107104 no beacon from osd.51 since 2020-10-29T03:38:40.689062-0400, 903.943390 seconds ago. marking down I haven't seen this happen before in any cluster. It's also strange that this only happens in this rack, the other two racks work fine. ID CLASS WEIGHT TYPE NAME -1 1545.35999 root default -206 515.12000 rack 206 -7 27.94499 host CEPH2-206-U16 ... -207 515.12000 rack 207 -17 27.94499 host CEPH2-207-U16 ... -208 515.12000 rack 208 -31 27.94499 host CEPH2-208-U16 ... That's how the CRUSHMap looks like. Straight forward and 3x replication over 3 racks. This issue only occurs in rack *207*. Has anybody seen this before or knows where to start? Wido

2 3

Bucket notification is working strange
by Krasaev 17 Nov '20

17 Nov '20

Hi everyone, I asked the same question in stackoverflow, but will repeat here. I configured bucket notification using a bucket owner creds and when the owner does actions I can see new events in a configured endpoint(kafka actually). However, when I try to do actions in the bucket, but with another user creds I do not see events in the configured notification topic. Is it expected behavior and each user has to configure own topic(is it possible if a user is not system at all)? Or I have missed something? Thank you. https://stackoverflow.com/questions/64384060/enable-bucket-notifications-fo…

2 1

OSD memory leak?
by Frank Schilder 16 Nov '20

16 Nov '20

Hi all, on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load. What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats: Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } } Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Am I looking at a memory leak here or are these heap stats expected? I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 25

monitor sst files continue growing
by Zhenshi Zhou 14 Nov '20

14 Nov '20

Hi all, My cluster is in wrong state. SST files in /var/lib/ceph/mon/xxx/store.db continue growing. It claims mon are using a lot of disk space. I set "mon compact on start = true" and restart one of the monitors. But it started and campacting for a long time, seems it has no end. [image: image.png]

4 14

Rados Crashing
by Brent Kennedy 12 Nov '20

12 Nov '20

We are performing file maintenance( deletes essentially ) and when the process gets to a certain point, all four rados gateways crash with the following: Log output: -5> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj verifying op params -4> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj pre-executing -3> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj executing -2> 2020-10-20 06:09:53.997 7f161758f700 10 monclient: get_auth_request con 0x55d2c02ff800 auth_method 0 -1> 2020-10-20 06:09:54.009 7f1609d74700 5 process_single_shard(): failed to acquire lock on obj_delete_at_hint.0000000079 0> 2020-10-20 06:09:54.035 7f15f1543700 -1 *** Caught signal (Segmentation fault) ** in thread 7f15f1543700 thread_name:civetweb-worker ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (()+0xf5d0) [0x7f161d3405d0] 2: (()+0x2bec80) [0x55d2bcd1fc80] 3: (std::string::assign(std::string const&)+0x2e) [0x55d2bcd2870e] 4: (rgw_bucket::operator=(rgw_bucket const&)+0x11) [0x55d2bce3e551] 5: (RGWObjManifest::obj_iterator::update_location()+0x184) [0x55d2bced7114] 6: (RGWObjManifest::obj_iterator::operator++()+0x263) [0x55d2bd092793] 7: (RGWRados::update_gc_chain(rgw_obj&, RGWObjManifest&, cls_rgw_obj_chain*)+0x51a) [0x55d2bd0939ea] 8: (RGWRados::Object::complete_atomic_modification()+0x83) [0x55d2bd093c63] 9: (RGWRados::Object::Delete::delete_obj()+0x74d) [0x55d2bd0a87ad] 10: (RGWDeleteObj::execute()+0x915) [0x55d2bd04b6d5] 11: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x915) [0x55d2bcdfbb35] 12: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x1cd8) [0x55d2bcdfdea8] 13: (RGWCivetWebFrontend::process(mg_connection*)+0x38e) [0x55d2bcd41a1e] 14: (()+0x36bace) [0x55d2bcdccace] 15: (()+0x36d76f) [0x55d2bcdce76f] 16: (()+0x36dc18) [0x55d2bcdcec18] 17: (()+0x7dd5) [0x7f161d338dd5] 18: (clone()+0x6d) [0x7f161c84302d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. My guess is that we need to add more resources to the gateways? They have 2 CPUs and 12GB of memory running as virtual machines on centOS 7.6 . Any thoughts? -Brent

2 2

The feasibility of mixed SSD and HDD replicated pool
by huww98＠outlook.com 10 Nov '20

10 Nov '20

Hi all, We are planning for a new pool to store our dataset using CephFS. These data are almost read-only (but not guaranteed) and consist of a lot of small files. Each node in our cluster has 1 * 1T SSD and 2 * 6T HDD, and we will deploy about 10 such nodes. We aim at getting the highest read throughput. If we just use a replicated pool of size 3 on SSD, we should get the best performance, however, that only leave us 1/3 of usable SSD space. And EC pools are not friendly to such small object read workload, I think. Now I’m evaluating a mixed SSD and HDD replication strategy. Ideally, I want 3 data replications, each on a different host (fail domain). 1 of them on SSD, the other 2 on HDD. And normally every read request is directed to SSD. So, if every SSD OSD is up, I’d expect the same read throughout as the all SSD deployment. I’ve read the documents and did some tests. Here is the crush rule I’m testing with: rule mixed_replicated_rule { id 3 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 1 type host step emit step take default class hdd step chooseleaf firstn -1 type host step emit } Now I have the following conclusions, but I’m not very sure: * The first OSD produced by crush will be the primary OSD (at least if I don’t change the “primary affinity”). So, the above rule is guaranteed to map SSD OSD as primary in pg. And every read request will read from SSD if it is up. * It is currently not possible to enforce SSD and HDD OSD to be chosen from different hosts. So, if I want to ensure data availability even if 2 hosts fail, I need to choose 1 SSD and 3 HDD OSD. That means setting the replication size to 4, instead of the ideal value 3, on the pool using the above crush rule. Am I correct about the above statements? How would this work from your experience? Thanks.

9 18

Ceph as a distributed filesystem and kerberos integration
by Alessandro Piazza 07 Nov '20

07 Nov '20

Dear all, I am experimenting with Ceph as a replacement for the AndrewFileSystem (https://en.wikipedia.org/wiki/Andrew_File_System) In my current setup, I am using AFS as a distributed filesystem for approximately 1000 users to store personal data and let them access their home directories and other shared data from multiple locations across different buildings. The authentication is managed by Kerberos (+ LDAP server). My goal is to replace AFS with CephFS but keep the current Kerberos database. Right now I've managed to set up a testing Ceph cluster with 6 nodes and 11 osds and I can mount CephFS using the kernel driver + CephX. However, from the Ceph docs, I can't understand if this might be a correct use-case for Ceph since the default authentication method CephX doesn't have a standard username/password authentication protocol. As far as I understand it requires the creation of a keyring with a random password generated on-the-fly which can then be used to mount the filesystem using the CephFS kernel module (https://docs.ceph.com/en/latest/cephfs/mount-using-kernel-driver/#mounting-…) As for the Kerberos integration, I found in the docs this page https://docs.ceph.com/en/latest/dev/ceph_krb_auth/ which is still a draft even if the last update was almost 2 years ago. From this page, I don't understand if the current version of Ceph supports full integration with GSSAPI/kerberos/LDAP. Since the docs only refer to keytab files, I was wondering if Kerberos can only be used as an authentication protocol between Ceph monitors/osds/metadata-servers and not for mounting the filesystem. Therefore I am asking - if anyone has tried Ceph for a similar use-case - what is the current status of Kerberos integration - if there are alternatives to CephX for mounting CephFS using kernel drivers which uses a username/password protocol Thank you and best regards, Alessandro Piazza

4 4

NoSuchKey on key that is visible in s3 list/radosgw bk
by Mariusz Gronczewski 06 Nov '20

06 Nov '20

Hi, I've got a problem on Octopus (15.2.3, debian packages) install, bucket S3 index shows a file: s3cmd ls s3://upvid/255/38355 --recursive 2020-07-27 17:48 50584342 s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 radosgw-admin bi list also shows it { "type": "plain", "idx": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "entry": { "name": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "instance": "", "ver": { "pool": 11, "epoch": 853842 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 50584342, "mtime": "2020-07-27T17:48:27.203008Z", "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", "storage_class": "", "owner": "filmweb-app", "owner_display_name": "filmweb app user", "content_type": "", "accounted_size": 50584342, "user_data": "", "appendable": "false" }, "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, but trying to download it via curl (I've set permissions to public0 only gets me <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>upvid</BucketName><RequestId>tx0000000000000000e716d-005f1f14cb-e478a-pl-war1</RequestId><HostId>e478a-pl-war1-pl</HostId></Error> (the actually nonexisting files shows access denied in same context) same with other tools: $ s3cmd get s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 /tmp download: 's3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' -> '/tmp/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' [1 of 1] ERROR: S3 error: 404 (NoSuchKey) cluster health is OK Any ideas what is happening here ? -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: admin(a)efigence.com

3 3

How to reset Log Levels
by Ml Ml 04 Nov '20

04 Nov '20

Hello, i played around with some log level i can´t remember and my logs are now getting bigger than my DVD-Movie collection. E.g.: journalctl -b -u ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)mon.ceph03.service > out.file is 1,1GB big. I did already try: ceph tell mon.ceph03 config set debug_mon 0/10 ceph tell mon.ceph03 config set debug_osd 0/10 ceph tell mon.ceph03 config set debug_mgr 0/10 ceph tell mon.ceph03 config set "mon_health_to_clog" false ceph tell mon.ceph03 config set "mon_health_log_update_period" 30 ceph tell mon.ceph03 config set "debug_mgr" "0/0" which made it better, but i really cant remember it all and would like to have the default values. Is there a way to reset those Log Values? Cheers, Michael

2 2

Monitor persistently out-of-quorum
by Ki Wong 02 Nov '20

02 Nov '20

Hello, I am at my wit's end. So I made a mistake in the configuration of my router and one of the monitors (out of 3) dropped out of the quorum and nothing I’ve done allow it to rejoin. That includes reinstalling the monitor with ceph-ansible. The connectivity issue is fixed. I’ve tested it using “nc” and the host can connect to both port 3300 and 6789 of the other monitors. But the wayward monitor continue to stay out of quorum. What is wrong? I see a bunch of “EBUSY” errors in the log, with the message: e1 handle_auth_request haven't formed initial quorum, EBUSY How do I fix this? Any help would be greatly appreciated. Many thanks, -kc With debug_mon at 1/10, I got these log snippets: 2020-10-28 15:40:05.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050353ec0 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050347ce0 after 2 seconds 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:07.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050347ce0 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050360660 after 2 seconds 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:09.107 7fb79253a700 -1 mon.mgmt03@2(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 1 at 2020-10-27 23:03:41.586915) 2020-10-28 15:40:09.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050360660 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x5640503606c0 after 2 seconds 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:11.962 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x5640503606c0 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050360900 after 2 seconds 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:12.354 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404cf25400 (start) method 2 payload 32 2020-10-28 15:40:12.354 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:40:12.354 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404cf25400 - … 2020-10-28 15:40:59.968 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:41:00.110 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404cae1000 (start) method 2 payload 22 2020-10-28 15:41:00.110 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.110 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404cae1000 - 2020-10-28 15:41:00.110 7fb793d3d700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404c8d4c00 (start) method 2 payload 22 2020-10-28 15:41:00.110 7fb793d3d700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.110 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404c8d4c00 - 2020-10-28 15:41:00.117 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404c630800 (start) method 2 payload 22 2020-10-28 15:41:00.117 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.117 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404c630800 - … 2020-10-28 15:42:42.379 7fb78d530700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2020-10-28 15:42:42.379 7fb78d530700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 60000.0 total, 600.0 interval Cumulative writes: 6 writes, 7 keys, 6 commit groups, 0.9 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 6 writes, 6 syncs, 0.86 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 3.02 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Sum 2/0 3.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Uptime(secs): 60000.0 total, 600.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count … 2020-10-28 17:17:11.781 7f5f694821c0 0 using public_addr v2:10.2.1.1:0/0 -> [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] 2020-10-28 17:17:11.781 7f5f694821c0 0 starting mon.mgmt03 rank -1 at public addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] at bind addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] mon_data /var/lib/ceph/mon/ceph-mgmt03 fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 preinit fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 initial_members mgmt01,mgmt02,mgmt03, filtering seed monmap 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 preinit clean up potentially inconsistent store state 2020-10-28 17:17:11.785 7f5f694821c0 0 -- [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] send_to message mon_probe(probe 374aed9e-5fc1-47e1-8d29-4416f7425e76 name mgmt03 new mon_release 14) v7 with empty dest 2020-10-28 17:17:13.191 7f5f5170d700 0 mon.mgmt03@-1(probing) e3 monmap addrs for rank 2 changed, i am [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0], monmap is v2:10.2.1.1:3300/0, respawning 2020-10-28 17:17:13.191 7f5f5170d700 -1 mon.mgmt03@-1(probing) e3 stashing newest monmap 3 for next startup 2020-10-28 17:17:13.191 7f5f5170d700 0 mon.mgmt03@-1(probing) e3 respawn 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 e: '/usr/bin/ceph-mon' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 0: '/usr/bin/ceph-mon' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 1: '-f' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 2: '--cluster' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 3: 'ceph' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 4: '--id' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 5: 'mgmt03' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 6: '--setuser' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 7: 'ceph' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 8: '--setgroup' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 9: 'ceph' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 respawning with exe /usr/bin/ceph-mon 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 exe_path /proc/self/exe 2020-10-28 17:17:13.217 7eff1f7cd1c0 0 ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable), process ceph-mon, pid 24265 2020-10-28 17:17:13.217 7eff1f7cd1c0 0 pidfile_write: ignore empty --pid-file 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 load: jerasure load: lrc load: isa 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option compression = kNoCompression 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option write_buffer_size = 33554432 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option compression = kNoCompression 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option write_buffer_size = 33554432 2020-10-28 17:17:13.246 7eff1f7cd1c0 1 rocksdb: do_open column families: [default] 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: RocksDB version: 6.1.2 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: Compile date Aug 13 2020 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: DB SUMMARY 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: CURRENT file: CURRENT 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: IDENTITY file: IDENTITY 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: MANIFEST file: MANIFEST-009877 size: 194 Bytes 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-mgmt03/store.db dir, Total Num: 2, files: 009874.sst 009876.sst 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-mgmt03/store.db: 009878.log size: 451 ; 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.error_if_exists: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.create_if_missing: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.paranoid_checks: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.env: 0x55beb27a5780 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.info_log: 0x55beb3d8e300 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_file_opening_threads: 16 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.statistics: (nil) 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_fsync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_log_file_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.log_file_time_to_roll: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.keep_log_file_num: 1000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.recycle_log_file_num: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_fallocate: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_mmap_reads: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_mmap_writes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_direct_reads: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.create_missing_column_families: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.db_log_dir: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-mgmt03/store.db 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_cache_numshardbits: 6 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_subcompactions: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_flushes: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.WAL_ttl_seconds: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.WAL_size_limit_MB: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.is_fd_close_on_exec: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.advise_random_on_open: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.db_write_buffer_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_buffer_manager: 0x55beb3d98b40 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_adaptive_mutex: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.rate_limiter: (nil) 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_recovery_mode: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_thread_tracking: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_pipelined_write: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.row_cache: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_filter: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_ingest_behind: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.preserve_deletes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.two_write_queues: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.manual_wal_flush: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.atomic_flush: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_jobs: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_compactions: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.delayed_write_rate : 16777216 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_total_wal_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_dump_period_sec: 600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_persist_period_sec: 600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_open_files: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bytes_per_sync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_bytes_per_sync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_readahead_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Compression algorithms supported: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZSTD supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kXpressCompression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kLZ4HCCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kLZ4Compression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kBZip2Compression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZlibCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kSnappyCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Fast CRC32 supported: Supported on x86 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-mgmt03/store.db/MANIFEST-009877 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.merge_operator: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_filter: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_filter_factory: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_factory: SkipListFactory 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_factory: BlockBasedTable 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55beb30daac0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55beb30f7010 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_buffer_size: 33554432 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_write_buffer_number: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression: NoCompression 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression: Disabled 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.prefix_extractor: nullptr 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.num_levels: 7 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.window_bits: -14 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.level: 32767 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.strategy: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.enabled: false 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.target_file_size_base: 67108864 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.target_file_size_multiplier: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.arena_block_size: 4194304 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.disable_auto_compactions: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_properties_collectors: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.inplace_update_support: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.inplace_update_num_locks: 10000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_huge_page_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bloom_locality: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_successive_merges: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.optimize_filters_for_hits: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.paranoid_file_checks: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.force_consistency_checks: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.report_bg_io_stats: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.ttl: 0 2020-10-28 17:17:13.248 7eff1f7cd1c0 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-mgmt03/store.db/MANIFEST-009877 succeeded,manifest_file_number is 9877, next_file_number is 9879, last_sequence is 2479, log_number is 9872,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 9872 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633249084, "job": 1, "event": "recovery_started", "log_files": [9878]} 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #9878 mode 2 2020-10-28 17:17:13.249 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633250074, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 9879, "file_size": 1336, "table_properties": {"data_size": 434, "index_size": 28, "filter_size": 69, "raw_key_size": 34, "raw_average_key_size": 34, "raw_value_size": 383, "raw_average_value_size": 383, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2020-10-28 17:17:13.249 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3036] Creating manifest 9880 2020-10-28 17:17:13.249 7eff1f7cd1c0 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2020-10-28 17:17:13.250 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633251509, "job": 1, "event": "recovery_finished"} 2020-10-28 17:17:13.253 7eff1f7cd1c0 4 rocksdb: DB pointer 0x55beb3d26400 2020-10-28 17:17:13.253 7eff05253700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2020-10-28 17:17:13.253 7eff05253700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 2.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 L6 1/0 2.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 5.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 2.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 L6 1/0 2.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 5.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2020-10-28 17:17:13.253 7eff1f7cd1c0 0 mon.mgmt03 does not exist in monmap, will attempt to join an existing cluster 2020-10-28 17:17:13.254 7eff1f7cd1c0 0 using public_addr v2:10.2.1.1:0/0 -> [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] 2020-10-28 17:17:13.254 7eff1f7cd1c0 0 starting mon.mgmt03 rank -1 at public addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] at bind addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] mon_data /var/lib/ceph/mon/ceph-mgmt03 fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 preinit fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 initial_members mgmt01,mgmt02,mgmt03, filtering seed monmap 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 preinit clean up potentially inconsistent store state 2020-10-28 17:17:13.258 7eff1f7cd1c0 0 -- [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] send_to message mon_probe(probe 374aed9e-5fc1-47e1-8d29-4416f7425e76 name mgmt03 new mon_release 14) v7 with empty dest

3 4

pgs stuck backfill_toofull
by Mark Johnson 02 Nov '20

02 Nov '20

I've been struggling with this one for a few days now. We had an OSD report as near full a few days ago. Had this happen a couple of times before and a reweight-by-utilization has sorted it out in the past. Tried the same again but this time we ended up with a couple of pgs in a state of backfill_toofull and a handful of misplaced objects as a result. Tried doing the reweight a few more times and it's been moving data around. We did have another osd trigger the near full alert but running the reweight a couple more times seems to have moved some of that data around a bit better. However, the original near_full osd doesn't seem to have changed much and the backfill_toofull pgs are still there. I'd keep doing the reweight-by-utilization but I'm not sure if I'm heading down the right path and if it will eventually sort it out. We have 14 pools, but the vast majority of data resides in just one of those pools (pool 20). The pgs in the backfill state are in pool 2 (as far as I can tell). That particular pool is used for some cephfs stuff and has a handful of large files in there (not sure if this is significant to the problem). All up, our utilization is showing as 55.13% but some of our OSDs are showing as 76% in use with this one problem sitting at 85.02%. Right now, I'm just not sure what the proper corrective action is. The last couple of reweights I've run have been a bit more targetted in that I've set it to only function on two OSDs at a time. If I run a test-reweight targetting only one osd, it does say it will reweight OSD 9 (the one at 85.02%). I gather this will move data away from this OSD and potentially get it below the threshold. However, at one point in the past couple of days, it's shown as no OSDs in a near full state, yet the two pgs in backfill_toofull didn't change. So, that's why I'm not sure continually reweighting is going to solve this issue. I'm a long way from knowledgable on Ceph so I'm not really sure what information is useful here. Here's a bit of info on what I'm seeing. Can provide anything else that might help. Basically, we have a three node cluster but only two have OSDs. The third is there simply to enable a quorum to be established. The OSDs are evenly spread across these two needs and the configuration of each is identical. We are running Jewel and are not in a position to upgrade at this stage. # ceph --version ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) # ceph health detail HEALTH_WARN 2 pgs backfill_toofull; 2 pgs stuck unclean; recovery 33/62099566 objects misplaced (0.000%); 1 near full osd(s) pg 2.52 is stuck unclean for 201822.031280, current state active+remapped+backfill_toofull, last acting [17,3] pg 2.18 is stuck unclean for 202114.617682, current state active+remapped+backfill_toofull, last acting [18,2] pg 2.18 is active+remapped+backfill_toofull, acting [18,2] pg 2.52 is active+remapped+backfill_toofull, acting [17,3] recovery 33/62099566 objects misplaced (0.000%) osd.9 is near full at 85% # ceph osd df ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 2 1.37790 1.00000 1410G 842G 496G 59.75 1.08 33 3 1.37790 0.45013 1410G 1079G 259G 76.49 1.39 21 4 1.37790 0.95001 1410G 1086G 253G 76.98 1.40 44 5 1.37790 1.00000 1410G 617G 722G 43.74 0.79 43 6 1.37790 0.65009 1410G 616G 722G 43.69 0.79 39 7 1.37790 0.95001 1410G 495G 844G 35.10 0.64 40 8 1.37790 1.00000 1410G 732G 606G 51.93 0.94 52 9 1.37790 0.70007 1410G 1199G 139G 85.02 1.54 37 10 1.37790 1.00000 1410G 611G 727G 43.35 0.79 41 11 1.37790 0.75006 1410G 495G 843G 35.11 0.64 32 0 1.37790 1.00000 1410G 731G 608G 51.82 0.94 43 12 1.37790 1.00000 1410G 851G 487G 60.36 1.09 44 13 1.37790 1.00000 1410G 378G 960G 26.82 0.49 38 14 1.37790 1.00000 1410G 969G 370G 68.68 1.25 37 15 1.37790 1.00000 1410G 724G 614G 51.35 0.93 35 16 1.37790 1.00000 1410G 491G 847G 34.84 0.63 43 17 1.37790 1.00000 1410G 862G 476G 61.16 1.11 50 18 1.37790 0.80005 1410G 1083G 255G 76.78 1.39 26 19 1.37790 0.65009 1410G 963G 375G 68.29 1.24 23 20 1.37790 1.00000 1410G 724G 614G 51.38 0.93 42 TOTAL 28219G 15557G 11227G 55.13 MIN/MAX VAR: 0.49/1.54 STDDEV: 15.57 # ceph pg ls backfill_toofull pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 2.18 9 0 0 18 0 0 3653 3653 active+remapped+backfill_toofull 2020-10-29 05:31:20.429912 610'549153 656:390372 [9,12] 9 [18,2] 18 594'547482 2020-10-25 20:28:39.680744 594'543841 2020-10-21 21:21:33.092868 2.52 15 0 0 15 0 0 4883 4883 active+remapped+backfill_toofull 2020-10-29 05:31:28.277898 652'502085 656:367288 [17,9] 17 [17,3] 17 594'499108 2020-10-26 11:06:48.417825 594'499108 2020-10-26 11:06:48.417825 pool : 17 18 19 11 20 21 12 13 0 14 1 15 2 16 | SUM -------------------------------------------------------------------------------------------------------------------------------- osd.4 3 0 0 0 9 2 0 0 12 1 9 0 7 1 | 44 osd.17 1 0 0 0 7 3 1 0 8 1 17 1 11 0 | 50 osd.18 0 0 0 0 9 0 0 0 4 0 7 0 5 0 | 25 osd.5 0 0 0 2 5 1 1 0 5 0 16 0 11 2 | 43 osd.6 0 1 0 1 5 2 0 0 9 0 13 1 7 0 | 39 osd.19 0 0 1 0 8 2 0 1 2 0 6 0 3 0 | 23 osd.7 0 0 0 0 4 1 1 0 3 0 12 0 19 0 | 40 osd.8 0 1 0 0 6 3 0 2 10 1 13 1 15 0 | 52 osd.9 1 0 2 0 10 2 0 0 4 1 6 1 10 0 | 37 osd.10 0 0 1 1 5 2 0 1 7 0 12 0 11 1 | 41 osd.20 1 0 0 0 6 1 0 1 7 0 8 1 17 0 | 42 osd.11 0 0 0 0 4 1 1 1 5 0 11 0 9 0 | 32 osd.12 0 0 1 1 7 1 0 0 5 1 12 1 14 1 | 44 osd.13 0 2 0 0 3 1 0 0 10 1 11 0 10 0 | 38 osd.0 0 1 0 1 6 3 0 1 7 0 11 0 13 0 | 43 osd.14 1 0 0 0 8 1 1 0 4 1 12 0 9 0 | 37 osd.15 1 0 2 1 6 1 1 0 8 0 7 0 6 2 | 35 osd.2 0 2 1 0 7 2 1 0 7 1 4 1 6 0 | 32 osd.3 0 0 0 0 9 0 0 0 2 0 4 0 5 0 | 20 osd.16 0 1 0 1 4 3 1 1 9 0 9 1 12 1 | 43 -------------------------------------------------------------------------------------------------------------------------------- SUM : 8 8 8 8 128 32 8 8 128 8 200 8 200 8 |

4 8

Fix PGs states
by Ing. Luis Felipe Domínguez Vega 02 Nov '20

02 Nov '20

Hi: I have this ceph status: ----------------------------------------------------------------------------- cluster: id: 039bf268-b5a6-11e9-bbb7-d06726ca4a78 health: HEALTH_WARN noout flag(s) set 1 osds down Reduced data availability: 191 pgs inactive, 2 pgs down, 35 pgs incomplete, 290 pgs stale 5 pgs not deep-scrubbed in time 7 pgs not scrubbed in time 327 slow ops, oldest one blocked for 233398 sec, daemons [osd.12,osd.36,osd.5] have slow ops. services: mon: 1 daemons, quorum fond-beagle (age 23h) mgr: fond-beagle(active, since 7h) osd: 48 osds: 45 up (since 95s), 46 in (since 8h); 4 remapped pgs flags noout data: pools: 7 pools, 2305 pgs objects: 350.37k objects, 1.5 TiB usage: 3.0 TiB used, 38 TiB / 41 TiB avail pgs: 6.681% pgs unknown 1.605% pgs not active 1835 active+clean 279 stale+active+clean 154 unknown 22 incomplete 10 stale+incomplete 2 down 2 remapped+incomplete 1 stale+remapped+incomplete -------------------------------------------------------------------------------------------- How can i fix all of unknown, incomplete, remmaped+incomplete, etc... i dont care if i need remove PGs

5 9

14.2.12 breaks mon_host pointing to Round Robin DNS entry
by Wido den Hollander 02 Nov '20

02 Nov '20

Hi, I already submitted a ticket: https://tracker.ceph.com/issues/47951 Maybe other people noticed this as well. Situation: - Cluster is running IPv6 - mon_host is set to a DNS entry - DNS entry is a Round Robin with three AAAA-records root@wido-standard-benchmark:~# ceph -s unable to parse addrs in 'mon.objects.xx.xxx.net' [errno 22] error connecting to the cluster root@wido-standard-benchmark:~# The relevant part of the ceph.conf: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx mon_host = mon.objects.xxx.xxx.xxx ms_bind_ipv6 = true This works fine with 14.2.11 and breaks under 14.2.12 Anybody else seeing this as well? Wido

5 5

OSD down, how to reconstruct it from its main and block.db parts ?
by Wladimir Mutel 30 Oct '20

30 Oct '20

Dear all, after breaking my experimental 1-host Ceph cluster and making one its pg 'incomplete' I left it in abandoned state for some time. Now I decided to bring it back into life and found that it can not start one of its OSDs (osd.1 to name it) "ceph osd df" shows : ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 113 MiB 4.7 GiB 1.1 TiB 59.77 0.69 102 up 1 hdd 2.84549 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 2 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 57 MiB 3.8 GiB 275 GiB 90.58 1.05 176 up 3 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 57 MiB 3.9 GiB 271 GiB 90.69 1.05 185 up 4 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 63 MiB 4.2 GiB 263 GiB 90.98 1.05 184 up 5 hdd 2.84549 1.00000 2.8 TiB 2.6 TiB 2.5 TiB 52 MiB 3.8 GiB 263 GiB 90.96 1.05 178 up 6 hdd 2.53400 1.00000 2.5 TiB 2.3 TiB 2.3 TiB 173 MiB 5.2 GiB 228 GiB 91.21 1.05 178 up 7 hdd 2.53400 1.00000 2.5 TiB 2.3 TiB 2.3 TiB 147 MiB 5.2 GiB 230 GiB 91.12 1.05 168 up TOTAL 19 TiB 17 TiB 16 TiB 662 MiB 31 GiB 2.6 TiB 86.48 MIN/MAX VAR: 0.69/1.05 STDDEV: 10.90 "ceph device ls" shows : DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY GIGABYTE_GP-ASACNE2100TTTDR_SN191108950380 p10s:nvme0n1 osd.1 osd.2 osd.3 osd.4 osd.5 WDC_WD30EFRX-68N32N0_WD-WCC7K1JJXVST p10s:sdd osd.1 WDC_WD30EFRX-68N32N0_WD-WCC7K1VUYPRA p10s:sda osd.6 WDC_WD30EFRX-68N32N0_WD-WCC7K2CKX8NT p10s:sdb osd.7 WDC_WD30EFRX-68N32N0_WD-WCC7K2UD8H74 p10s:sde osd.2 WDC_WD30EFRX-68N32N0_WD-WCC7K2VFTR1F p10s:sdh osd.5 WDC_WD30EFRX-68N32N0_WD-WCC7K3CYKL87 p10s:sdf osd.3 WDC_WD30EFRX-68N32N0_WD-WCC7K6FPZAJP p10s:sdc osd.0 WDC_WD30EFRX-68N32N0_WD-WCC7K7FXSCRN p10s:sdg osd.4 In my last migration, I created a bluestore volume with external block.db like this : "ceph-volume lvm prepare --bluestore --data /dev/sdd1 --block.db /dev/nvme0n1p4" And I can see this metadata by "ceph-bluestore-tool show-label --dev /dev/ceph-e53b65ba-5eb0-44f5-9160-a2328f787a0f/osd-block-8c6324a3-0364-4fad-9dcb-81a1661ee202" : { "/dev/ceph-e53b65ba-5eb0-44f5-9160-a2328f787a0f/osd-block-8c6324a3-0364-4fad-9dcb-81a1661ee202": { "osd_uuid": "8c6324a3-0364-4fad-9dcb-81a1661ee202", "size": 3000588304384, "btime": "2020-07-12T11:34:16.579735+0300", "description": "main", "bfm_blocks": "45785344", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "65536", "bfm_size": "3000588304384", "bluefs": "1", "ceph_fsid": "49cdfe90-6f6e-4afe-8558-bf14a13aadfa", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQD9ygpf+7+MABAAqtj4y1YYgxwCaAN/jgDSwg==", "ready": "ready", "require_osd_release": "14", "whoami": "1" } } and by "ceph-bluestore-tool show-label --dev /dev/nvme0n1p4" : { "/dev/nvme0n1p4": { "osd_uuid": "8c6324a3-0364-4fad-9dcb-81a1661ee202", "size": 128025886720, "btime": "2020-07-12T11:34:16.592054+0300", "description": "bluefs db" } } As you see, their osd_uuid is equal. But when I try to start it by hand : "systemctl restart ceph-osd@1" , I get this in the logs : ("journalctl -b -u ceph-osd@1") -- Logs begin at Tue 2020-10-13 19:09:49 EEST, end at Fri 2020-10-23 16:59:38 EEST. -- жов 23 16:59:36 p10s systemd[1]: Starting Ceph object storage daemon osd.1... жов 23 16:59:36 p10s systemd[1]: Started Ceph object storage daemon osd.1. жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 AuthRegistry(0x560776222940) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.943+0300 7f513cebedc0 -1 AuthRegistry(0x560776222940) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 AuthRegistry(0x7fff46ea5d80) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: 2020-10-23T16:59:36.947+0300 7f513cebedc0 -1 AuthRegistry(0x7fff46ea5d80) no keyring found at /var/lib/ceph/osd/ceph-1/keyring, disabling cephx жов 23 16:59:36 p10s ceph-osd[3987]: failed to fetch mon config (--no-mon-config to skip) жов 23 16:59:36 p10s systemd[1]: ceph-osd(a)1.service: Main process exited, code=exited, status=1/FAILURE жов 23 16:59:36 p10s systemd[1]: ceph-osd(a)1.service: Failed with result 'exit-code'. And so my question is, how to make this OSD known again to Ceph cluster without recreating it anew with ceph-volume ? I see that every folder under "/var/lib/ceph/osd/" is a tmpfs mount point filled with appropriate files and symlinks, except of "/var/lib/ceph/osd/ceph-1", which is just an empty folder not mounted anywhere. I tried to run "ceph-bluestore-tool prime-osd-dir --dev /dev/ceph-e53b65ba-5eb0-44f5-9160-a2328f787a0f/osd-block-8c6324a3-0364-4fad-9dcb-81a1661ee202 --path /var/lib/ceph/osd/ceph-1" it created some files under /var/lib/ceph/osd/ceph-1 but without tmpfs mount, and these files belonged to root. I changed owner of these files into ceph.ceph , I created appropriate symlinks for block and block.db but ceph-osd@1 did not want to start either. Only "unable to find keyring" messages disappeared. Please give any help on where to move next. Thanks in advance for your help.

2 4

MDS_CLIENT_LATE_RELEASE: 3 clients failing to respond to capability release
by Frank Schilder 30 Oct '20

30 Oct '20

Dear cephers, I have a somewhat strange situation. I have the health warning: # ceph health detail HEALTH_WARN 3 clients failing to respond to capability release MDS_CLIENT_LATE_RELEASE 3 clients failing to respond to capability release mdsceph-12(mds.0): Client sn106.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 30716617 mdsceph-12(mds.0): Client sn269.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 30717358 mdsceph-12(mds.0): Client sn009.hpc.ait.dtu.dk:con-fs2-hpc failing to respond to capability release client_id: 30749150 However, these clients are not busy right now. Also, they hold almost nothing; see snippets from "session ls" below. It is possible that a very IO intensive application was running on these nodes and these release requests got stuck. How do I resolve this issue? Can I just evict the client? Version is mimic 13.2.8. Note that we execute a drop cache command after a job finishes on these clients. Its possible that the clients dropped the caps already before the MDS request was handled/received. Best regards, Frank { "id": 30717358, "num_leases": 0, "num_caps": 44, "state": "open", "request_load_avg": 0, "uptime": 6632206.332307, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, "inst": "client.30717358 192.168.57.140:0/3212676185", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn269.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/home" } }, -- { "id": 30716617, "num_leases": 0, "num_caps": 48, "state": "open", "request_load_avg": 1, "uptime": 6632206.336307, "replay_requests": 0, "completed_requests": 1, "reconnecting": false, "inst": "client.30716617 192.168.56.233:0/2770977433", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn106.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/home" } }, -- { "id": 30749150, "num_leases": 0, "num_caps": 44, "state": "open", "request_load_avg": 0, "uptime": 6632206.338307, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, "inst": "client.30749150 192.168.56.136:0/2578719015", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn009.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/home" } }, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 3

Very high read IO during backfilling
by Kamil Szczygieł 30 Oct '20

30 Oct '20

Hi, We're running Octopus and we've 3 control plane nodes (12 core, 64 GB memory each) that are running mon, mds and mgr and also 4 data nodes (12 core, 256 GB memory, 13x10TB HDDs each). We've increased number of PGs inside our pool, which resulted in all OSDs going crazy and reading the average of 900 M/s constantly (based on iotop). This has resulted in slow ops and very low recovery speed. Any tips on how to handle this kind of situation? We've osd_recovery_sleep_hdd set to 0.2, osd_recovery_max_active set to 5 and osd_max_backfills set to 4. Some OSDs are reporting slow ops constantly and iowait on machines is at 70-80% constantly.

3 2

RBD low iops with 4k object size
by w1kl4s 30 Oct '20

30 Oct '20

Hello! I'm currently in process of playing around with ceph, and before migrating my servers to use it i wanted to benchmark it to have some idea what expected performance it might achieve. I am using 2 400GB Intel S3700 ssds with crush rule created as follows: ➜ ~ ceph osd crush rule create-replicated ssd_mirror default osd ssd And later creating pool ssd_test setting both size and min_size to 2. I proceeded to benchmark it with rbd bench, and using default image object size of 4M the result is about what i have been expecting (10k iops per OSD). However, when creating image with 4K block size, IOPS go down the drain, despite theoretically being better suited for it (4k writes should be amplified to 4M ones in case of 4M object size, reducing iops, while 4k object size should be the one without write amplification). Here is what i used to create the images and benchmark them: ➜ ~ rbd create -p ssd_test --size 16G --object-size 4K big_4k ➜ ~ rbd create -p ssd_test --size 16G --object-size 4M big_4m ➜ ~ rbd bench --io-type write --io-size 4K --io-pattern rand --io-total 256M ssd_test/big_4m bench type write io_size 4096 io_threads 16 bytes 268435456 pattern random SEC OPS OPS/SEC BYTES/SEC 1 8624 8640.00 35389458.37 2 12928 6472.00 26509325.75 3 22672 7562.67 30976698.73 4 37344 9340.00 38256659.85 5 52000 10403.21 42611529.30 elapsed: 6 ops: 65536 ops/sec: 10886.38 bytes/sec: 44590630.43 ➜ ~ rbd bench --io-type write --io-size 4K --io-pattern rand --io-total 256M ssd_test/big_4k bench type write io_size 4096 io_threads 16 bytes 268435456 pattern random SEC OPS OPS/SEC BYTES/SEC 1 6736 6645.67 27220675.51 2 7312 3664.00 15007751.77 3 7920 2645.33 10835290.95 4 8560 2135.46 8746841.19 5 9168 1828.03 7487596.24 6 9760 605.77 2481232.06 7 10368 610.22 2499477.33 8 10960 605.58 2480447.50 9 11536 596.15 2441847.42 10 12160 599.84 2456944.34 11 12816 610.22 2499477.33 12 13488 624.00 2555905.32 13 14144 636.29 2606249.15 14 14816 654.43 2680544.09 15 15472 661.87 2711022.99 16 16128 664.53 2721901.89 17 16800 662.40 2713191.81 18 17472 668.27 2737248.00 19 18080 650.72 2665341.09 20 18544 614.40 2516583.70 21 19056 584.20 2392875.94 22 19376 513.97 2105207.79 23 19584 416.73 1706936.95 24 19840 343.21 1405804.16 25 19984 287.77 1178705.65 26 20128 214.06 876780.01 27 20368 199.04 815255.64 28 20896 265.80 1088726.66 29 21488 339.65 1391222.48 30 22080 419.87 1719795.76 31 22688 512.82 2100513.91 32 23280 582.40 2385511.64 33 23888 598.88 2453010.08 34 24480 600.32 2458916.20 35 25104 603.83 2473304.79 36 25744 611.69 2505480.88 37 26432 629.39 2577994.94 38 27072 636.80 2608334.15 39 27760 653.39 2676272.30 40 28464 673.08 2756924.50 41 29104 673.08 2756924.50 42 29776 668.27 2737216.44 43 30400 665.60 2726299.01 44 31088 667.74 2735051.18 45 31744 654.43 2680544.09 46 32400 657.10 2691471.89 47 33056 657.05 2691283.44 48 33680 655.48 2684829.53 49 34304 642.69 2632442.61 50 35024 656.53 2689128.69 51 35616 645.27 2643006.18 52 36240 635.78 2604167.48 53 36880 641.03 2625642.38 54 37568 652.28 2671732.80 55 38192 634.62 2599385.96 56 38832 642.69 2632442.61 57 39472 647.44 2651898.81 58 40144 651.24 2667468.26 59 40816 648.56 2656512.56 60 41424 647.96 2654025.43 61 42112 653.39 2676272.30 62 42752 655.48 2684829.53 63 43392 649.08 2658636.07 64 44064 650.12 2662893.29 65 44704 653.91 2678406.49 66 45376 654.37 2680302.91 67 46048 658.67 2697926.26 68 46704 663.46 2717539.87 69 47344 654.95 2682685.09 70 48000 659.73 2702246.39 71 48672 660.26 2704411.65 72 49344 659.73 2702246.39 73 50032 665.07 2724119.71 74 50688 672.03 2752618.79 75 51344 667.20 2732847.38 76 52000 661.89 2711116.75 77 52592 650.12 2662893.29 78 53248 642.69 2632442.61 79 53920 645.37 2643426.29 80 54576 649.52 2660425.81 81 55232 649.00 2658288.92 82 55872 655.48 2684829.52 83 56544 661.32 2708752.60 84 57184 653.32 2676010.99 85 57856 656.00 2686977.39 86 58512 657.05 2691283.44 87 59216 668.27 2737216.44 88 59904 670.39 2745923.21 89 60592 679.97 2785150.68 90 61264 680.51 2787375.24 91 61936 684.25 2802700.09 92 62608 680.58 2787648.31 93 63248 667.73 2735030.17 94 63856 654.90 2682454.03 95 64496 645.88 2645539.34 96 65184 648.05 2654392.43 elapsed: 105 ops: 65536 ops/sec: 621.88 bytes/sec: 2547213.95 What could be causing such decrease in random write performance in case of 4k object size image? Best regards, w1kl4s

1 0

Corrupted RBD image
by Ing. Luis Felipe Domínguez Vega 30 Oct '20

30 Oct '20

Hi: I tried get info from a RBD image but: ------------------------------------------------------------------------- root@fond-beagle:/# rbd list --pool cinder-ceph | grep volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda > volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda root@fond-beagle:/# rbd info --pool cinder-ceph volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda > rbd: error opening image volume-dfcca6c8-cb96-4b79-bc85-b200a061dcda: > (2) No such file or directory ---------------------------------------------------------------------- THis is that the metadata show the image but the content was removed?

3 3

Mon crashes when adding 4th OSD
by Lalit Maganti 30 Oct '20

30 Oct '20

(sending this email again as the first time was blocked because my attached log file was too big) Hi all, *Context*: I'm running Ceph Octopus 15.2.5 (the latest as of this email) using Rook on a toy Kubernetes cluster of two nodes. I've got a single Ceph mon node running perfectly with 3 OSDs . There are two pools running which were created as part of a CephFS install. *Problem*: when I try to add my 4th OSD, the Ceph mon starts crashing on the OSDMonitor::build_incremental function. I've checked on the mailing lists and just in general and the last instance of this issue seems to have been 7 years ago so I'm probably not hitting the same thing! *Question*: I was wondering if anyone had ideas on what I might be doing wrong? I'm very new to Ceph so my suspicion is that it's something to do with my configuration but given I'm literally just adding an OSD and everything is fine otherwise, I'm not sure what my mistake might be. Please find the bug I filed on the Ceph tracker here <https://tracker.ceph.com/issues/48026> where I've provided a mon log file with log level 20. Kind regards, Lalit Maganti

1 0

bluefs mount failed(crash) after a long time
by Elians Wan 30 Oct '20

30 Oct '20

Anyone can help? Bluefs mount failed after a long time The error message: 2020-10-30 05:33:54.906725 7f1ad73f5e00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-30/block size 7.28TiB 2020-10-30 05:33:54.906758 7f1ad73f5e00 1 bluefs mount 2020-10-30 06:00:32.881850 7f1ad73f5e00 -1 *** Caught signal (Segmentation fault) ** in thread 7f1ad73f5e00 thread_name:ceph-osd ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable) 1: (()+0xaa2044) [0x5570d12af044] 2: (()+0x11390) [0x7f1ad56d2390] 3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xad4) [0x5570d125ea34] 4: (BlueFS::_replay(bool)+0x409) [0x5570d1267599] 5: (BlueFS::mount()+0x209) [0x5570d126b659] 6: (BlueStore::_open_db(bool)+0x169c) [0x5570d117acdc] 7: (BlueStore::_mount(bool)+0x3ad) [0x5570d11aeded] 8: (OSD::init()+0x3e2) [0x5570d0d00f12] 9: (main()+0x2f0a) [0x5570d0c0a0ca] 10: (__libc_start_main()+0xf0) [0x7f1ad4658830] 11: (_start()+0x29) [0x5570d0c97329] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2 1

MDS restarts after enabling msgr2
by Stefan Kooman 30 Oct '20

30 Oct '20

Hi List, After a successful upgrade from Mimic 13.2.8 to Nautilus 14.2.12 we enabled msgr2. Soon after that both of the MDS servers (active / active-standby) restarted. We did not hit any ASSERTS this time, so that's good :>. However, I have not seen this happening on four different test clusters (while running a slightly older Nautilus release), so I certainly did not expect that. Most of the connections switched over to 3300 (apart from the cephfs kernel clients) and that all kept on working. Anybody else has seen this behavior before? Gr. Stefan

2 1

frequent Monitor down
by Andrei Mikhailovsky 30 Oct '20

30 Oct '20

Hello everyone, I am having regular messages that the Monitors are going down and up: 2020-10-27T09:50:49.032431+0000 mon .arh-ibstorage2-ib ( mon .1) 2248 : cluster [WRN] Health check failed: 1/4 mons down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) 2020-10-27T09:50:49.123511+0000 mon .arh-ibstorage2-ib ( mon .1) 2250 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout flag(s) set; 43 pgs not deep-scrubbed in time; 12 pgs not scrubbed in time 2020-10-27T09:50:52.735457+0000 mon .arh-ibstorage1-ib ( mon .0) 31287 : cluster [INF] Health check cleared: MON_DOWN (was: 1/4 mons down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib) 2020-10-27T12:35:20.556458+0000 mon .arh-ibstorage2-ib ( mon .1) 2260 : cluster [WRN] Health check failed: 1/4 mons down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) 2020-10-27T12:35:20.643282+0000 mon .arh-ibstorage2-ib ( mon .1) 2262 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout flag(s) set; 47 pgs not deep-scrubbed in time; 14 pgs not scrubbed in time This happens on a daily basis several times a day. Could you please let me know how to fix this annoying problem? I am running ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) on Ubuntu 18.04 LTS with latest updates. Thanks Andrei

6 8

Huge HDD ceph monitor usage
by Ing. Luis Felipe Domínguez Vega 29 Oct '20

29 Oct '20

How can i free the store of ceph monitor?: ------------------------------------------------------------------------ root@fond-beagle:/var/lib/ceph/mon/ceph-fond-beagle# du -h -d1 542G ./store.db 542G . ------------------------------------------------------------------------

5 23

dashboard object gateway not working
by Siegfried Höllrigl 29 Oct '20

29 Oct '20

We are running Ceph 14.2.12 and would like to manage our object gateways via the dashboard. The rados gateways are running on different (virtual) machines than the mon servers (where mon, mgr and mds are running). The dashboard seems to be running fine. But when we cklick on "Object Gateway" we get the following error message : Sorry, we could not find what you were looking for 500 - Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. 10/27/20 12:30:32 PM The logfiles (of mgr and radosgw) do not show anything helpful. Maybe we have missed someting that needs to be enabled/installed/activated ?!? Any Ideas ?

2 4

Cloud Sync Module
by Sailaja Yedugundla 29 Oct '20

29 Oct '20

I am trying to configure cloud sync module in my ceph cluster to implement backup to AWS S3 cluster. I could not find configure using the available documentation. Can someone help me to implement this? Thanks, Sailaja

1 0

Ceph User Survey 2020 - Working Group Invite
by anantha.adiga＠intel.com 28 Oct '20

28 Oct '20

Hello all, This is an invite to all interested to join a working group being formed for 2020 Ceph User Survey planning. The focus is to augment the questionnaire coverage, explore survey delivery formats and to expand the survey reach to audience across the world. The popularity and adoption of Ceph is growing steadily and so are the deployment options. Survey feedback has certainly helped the community to focus on user's asks and make Ceph better for their needs. This time we want to make the survey experience more enriching to the community of developers and to the user community. As a sample, here are a few questions that have been collected to help build better hardware options for Ceph across Enterprise and CSPs. The working group can help refine them for content and importance. There will be questions in other categories like Ceph configuration that members will help assess for relevance and inclusion and find some innovative approaches to reach boarder audience. Do you prefer single socket servers for Ceph OSD nodes? Which drive form factors are important to you (NVMe drives: U.2 or ruler (E1.L))? How many drives per server fits your need? What drive capacities are important to you? Do you separate metadata and data on different classes of media? Do you use Optane 3D XPoint or NAND for BlueStore metadata? Which caching method, client side vs OSD side is more useful to you? As always, many many thanks to Mike Perez who is driving the user survey effort and making it better every passing year. Thank you, Anantha Adiga

3 2

Re: Huge HDD ceph monitor usage [EXT]
by Ing. Luis Felipe Domínguez Vega 27 Oct '20

27 Oct '20

Well. 7 hosts up and recovery start and stop in 3 hours more or less, now the cluster is not recovering any more... can be that needs more hosts? El 2020-10-27 13:58, Eugen Block escribió: > Hm, that would be new to me that the mgr service is required for > recovery, but maybe I missed something and it is a crucial component > meanwhile, I'm not sure. > > The large MON db could be due to a large backlog of cluster maps. The > more and the longer a recovery is taking the more space the MONs > require, that's unfortunate. I think there are ways to trim that > without a major impact but I don't know how, never had to do that. But > during recovery progress the usage should decrease, I believe. > > > Zitat von "Ing. Luis Felipe Domínguez Vega" <luis.dominguez(a)desoft.cu>: > >> One thing, only when i started the ceph-mgr is that the cluster began >> to recover... I added the 2 new hosts and now is recovering, but i >> dont known why ceph monitor is consuming 500 GB of HDD.... >> >> El 2020-10-27 09:59, Eugen Block escribió: >>> Your pool 'data_storage' has a size of 7 (or 7 chunks since it's >>> erasure-coded) and the rule requires each chunk on a different host >>> but you currently have only 5 hosts available, that's why the >>> recovery >>> is not progressing. It's waiting for two more hosts. Unfortunately, >>> you can't change the EC profile or the rule of that pool. I'm not >>> sure >>> if it would work in the current cluster state, but if you can't add >>> two more hosts (which would be your best option for recovery) it >>> might >>> be possible to create a new replicated pool (you seem to have enough >>> free space) and copy the contents from that EC pool. But as I said, >>> I'm not sure if that would work in a degraded state, I've never tried >>> that. >>> >>> So your best bet is to get two more hosts somehow. >>> >>> >>>> pool 4 'data_storage' erasure profile desoft size 7 min_size 5 >>>> crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 >>>> autoscale_mode off last_change 154384 lfor 0/121016/121014 flags >>>> hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 16384 >>>> application rbd >>> >>> >>> Zitat von "Ing. Luis Felipe Domínguez Vega" >>> <luis.dominguez(a)desoft.cu>: >>> >>>> Needed data: >>>> >>>> ceph -s : https://pastebin.ubuntu.com/p/S9gKjyZtdK/ >>>> ceph osd tree : https://pastebin.ubuntu.com/p/SCZHkk6Mk4/ >>>> ceph osd df : (later, because i'm waiting since 10 >>>> minutes and not output yet) >>>> ceph osd pool ls detail : https://pastebin.ubuntu.com/p/GRdPjxhv3D/ >>>> crush rules : (ceph osd crush rule dump) >>>> https://pastebin.ubuntu.com/p/cjyjmbQ4Wq/ >>>> >>>> El 2020-10-27 07:14, Eugen Block escribió: >>>>>> I understand, but i delete the OSDs from CRUSH map, so ceph don't >>>>>> wait for these OSDs, i'm right? >>>>> >>>>> It depends on your actual crush tree and rules. Can you share >>>>> (maybe >>>>> you already did) >>>>> >>>>> ceph osd tree >>>>> ceph osd df >>>>> ceph osd pool ls detail >>>>> >>>>> and a dump of your crush rules? >>>>> >>>>> As I already said, if you have rules in place that distribute data >>>>> across 2 DCs and one of them is down the PGs will never recover >>>>> even >>>>> if you delete the OSDs from the failed DC. >>>>> >>>>> >>>>> >>>>> Zitat von "Ing. Luis Felipe Domínguez Vega" >>>>> <luis.dominguez(a)desoft.cu>: >>>>> >>>>>> I understand, but i delete the OSDs from CRUSH map, so ceph don't >>>>>> wait for these OSDs, i'm right? >>>>>> >>>>>> El 2020-10-27 04:06, Eugen Block escribió: >>>>>>> Hi, >>>>>>> >>>>>>> just to clarify so I don't miss anything: you have two DCs and >>>>>>> one of >>>>>>> them is down. And two of the MONs were in that failed DC? Now you >>>>>>> removed all OSDs and two MONs from the failed DC hoping that your >>>>>>> cluster will recover? If you have reasonable crush rules in place >>>>>>> (e.g. to recover from a failed DC) your cluster will never >>>>>>> recover in >>>>>>> the current state unless you bring OSDs back up on the second DC. >>>>>>> That's why you don't see progress in the recovery process, the >>>>>>> PGs are >>>>>>> waiting for their peers in the other DC so they can follow the >>>>>>> crush >>>>>>> rules. >>>>>>> >>>>>>> Regards, >>>>>>> Eugen >>>>>>> >>>>>>> >>>>>>> Zitat von "Ing. Luis Felipe Domínguez Vega" >>>>>>> <luis.dominguez(a)desoft.cu>: >>>>>>> >>>>>>>> I was 3 mons, but i have 2 physical datacenters, one of them >>>>>>>> breaks with not short term fix, so i remove all osds and ceph >>>>>>>> mon (2 of them) and now i have only the osds of 1 datacenter >>>>>>>> with the monitor. I was stopped the ceph manager, but i was >>>>>>>> see that when i restart a ceph manager then ceph -s show >>>>>>>> recovering info for a short term of 20 min more or less, then >>>>>>>> dissapear all info. >>>>>>>> >>>>>>>> The thing is that sems the cluster is not self recovering and >>>>>>>> the ceph monitor is "eating" all of the HDD. >>>>>>>> >>>>>>>> El 2020-10-26 15:57, Eugen Block escribió: >>>>>>>>> The recovery process (ceph -s) is independent of the MGR >>>>>>>>> service but >>>>>>>>> only depends on the MON service. It seems you only have the one >>>>>>>>> MON, >>>>>>>>> if the MGR is overloading it (not clear why) it could help to >>>>>>>>> leave >>>>>>>>> MGR off and see if the MON service then has enough RAM to >>>>>>>>> proceed with >>>>>>>>> the recovery. Do you have any chance to add two more MONs? A >>>>>>>>> single >>>>>>>>> MON is of course a single point of failure. >>>>>>>>> >>>>>>>>> >>>>>>>>> Zitat von "Ing. Luis Felipe Domínguez Vega" >>>>>>>>> <luis.dominguez(a)desoft.cu>: >>>>>>>>> >>>>>>>>>> El 2020-10-26 15:16, Eugen Block escribió: >>>>>>>>>>> You could stop the MGRs and wait for the recovery to finish, >>>>>>>>>>> MGRs are >>>>>>>>>>> not a critical component. You won’t have a dashboard or >>>>>>>>>>> metrics >>>>>>>>>>> during/of that time but it would prevent the high RAM usage. >>>>>>>>>>> >>>>>>>>>>> Zitat von "Ing. Luis Felipe Domínguez Vega" >>>>>>>>>>> <luis.dominguez(a)desoft.cu>: >>>>>>>>>>> >>>>>>>>>>>> El 2020-10-26 12:23, 胡玮文 escribió: >>>>>>>>>>>>>> 在 2020年10月26日，23:29，Ing. Luis Felipe Domínguez Vega >>>>>>>>>>>>>> <luis.dominguez(a)desoft.cu> 写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>> mgr: fond-beagle(active, since 39s) >>>>>>>>>>>>> >>>>>>>>>>>>> Your manager seems crash looping, it only started since >>>>>>>>>>>>> 39s. Looking >>>>>>>>>>>>> at mgr logs may help you identify why your cluster is not >>>>>>>>>>>>> recovering. >>>>>>>>>>>>> You may hit some bug in mgr. >>>>>>>>>>>> Noup, I'm restarting the ceph manager because they eat all >>>>>>>>>>>> server RAM and then i have an script that when i have >>>>>>>>>>>> 1GB of Free Ram (the server has 94 Gb of RAM) then >>>>>>>>>>>> restart the manager, i dont known why and the logs of >>>>>>>>>>>> manager are: >>>>>>>>>>>> >>>>>>>>>>>> ----------------------------------- >>>>>>>>>>>> root@fond-beagle:/var/lib/ceph/mon/ceph-fond-beagle/store.db# >>>>>>>>>>>> tail -f /var/log/ceph/ceph-mgr.fond-beagle.log >>>>>>>>>>>> 2020-10-26T12:54:12.497-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) log [DBG] : pgmap v584: 2305 pgs: 4 >>>>>>>>>>>> active+undersized+degraded+remapped, 4 >>>>>>>>>>>> active+recovery_unfound+undersized+degraded+remapped, 2104 >>>>>>>>>>>> active+clean, 5 active+undersized+degraded, 34 >>>>>>>>>>>> incomplete, 154 unknown; 1.7 TiB data, 2.9 TiB used, >>>>>>>>>>>> 21 TiB / 24 TiB avail; 347248/2606900 objects degraded >>>>>>>>>>>> (13.320%); 107570/2606900 objects misplaced (4.126%); >>>>>>>>>>>> 19/404328 objects unfound (0.005%) >>>>>>>>>>>> 2020-10-26T12:54:12.497-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) do_log log to syslog >>>>>>>>>>>> 2020-10-26T12:54:14.501-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) log [DBG] : pgmap v585: 2305 pgs: 4 >>>>>>>>>>>> active+undersized+degraded+remapped, 4 >>>>>>>>>>>> active+recovery_unfound+undersized+degraded+remapped, 2104 >>>>>>>>>>>> active+clean, 5 active+undersized+degraded, 34 >>>>>>>>>>>> incomplete, 154 unknown; 1.7 TiB data, 2.9 TiB used, >>>>>>>>>>>> 21 TiB / 24 TiB avail; 347248/2606900 objects degraded >>>>>>>>>>>> (13.320%); 107570/2606900 objects misplaced (4.126%); >>>>>>>>>>>> 19/404328 objects unfound (0.005%) >>>>>>>>>>>> 2020-10-26T12:54:14.501-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) do_log log to syslog >>>>>>>>>>>> 2020-10-26T12:54:16.517-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) log [DBG] : pgmap v586: 2305 pgs: 4 >>>>>>>>>>>> active+undersized+degraded+remapped, 4 >>>>>>>>>>>> active+recovery_unfound+undersized+degraded+remapped, 2104 >>>>>>>>>>>> active+clean, 5 active+undersized+degraded, 34 >>>>>>>>>>>> incomplete, 154 unknown; 1.7 TiB data, 2.9 TiB used, >>>>>>>>>>>> 21 TiB / 24 TiB avail; 347248/2606900 objects degraded >>>>>>>>>>>> (13.320%); 107570/2606900 objects misplaced (4.126%); >>>>>>>>>>>> 19/404328 objects unfound (0.005%) >>>>>>>>>>>> 2020-10-26T12:54:16.517-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) do_log log to syslog >>>>>>>>>>>> 2020-10-26T12:54:18.521-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) log [DBG] : pgmap v587: 2305 pgs: 4 >>>>>>>>>>>> active+undersized+degraded+remapped, 4 >>>>>>>>>>>> active+recovery_unfound+undersized+degraded+remapped, 2104 >>>>>>>>>>>> active+clean, 5 active+undersized+degraded, 34 >>>>>>>>>>>> incomplete, 154 unknown; 1.7 TiB data, 2.9 TiB used, >>>>>>>>>>>> 21 TiB / 24 TiB avail; 347248/2606900 objects degraded >>>>>>>>>>>> (13.320%); 107570/2606900 objects misplaced (4.126%); >>>>>>>>>>>> 19/404328 objects unfound (0.005%) >>>>>>>>>>>> 2020-10-26T12:54:18.521-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) do_log log to syslog >>>>>>>>>>>> 2020-10-26T12:54:20.537-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) log [DBG] : pgmap v588: 2305 pgs: 4 >>>>>>>>>>>> active+undersized+degraded+remapped, 4 >>>>>>>>>>>> active+recovery_unfound+undersized+degraded+remapped, 2104 >>>>>>>>>>>> active+clean, 5 active+undersized+degraded, 34 >>>>>>>>>>>> incomplete, 154 unknown; 1.7 TiB data, 2.9 TiB used, >>>>>>>>>>>> 21 TiB / 24 TiB avail; 347248/2606900 objects degraded >>>>>>>>>>>> (13.320%); 107570/2606900 objects misplaced (4.126%); >>>>>>>>>>>> 19/404328 objects unfound (0.005%) >>>>>>>>>>>> 2020-10-26T12:54:20.537-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) do_log log to syslog >>>>>>>>>>>> 2020-10-26T12:54:22.541-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) log [DBG] : pgmap v589: 2305 pgs: 4 >>>>>>>>>>>> active+undersized+degraded+remapped, 4 >>>>>>>>>>>> active+recovery_unfound+undersized+degraded+remapped, 2104 >>>>>>>>>>>> active+clean, 5 active+undersized+degraded, 34 >>>>>>>>>>>> incomplete, 154 unknown; 1.7 TiB data, 2.9 TiB used, >>>>>>>>>>>> 21 TiB / 24 TiB avail; 347248/2606900 objects degraded >>>>>>>>>>>> (13.320%); 107570/2606900 objects misplaced (4.126%); >>>>>>>>>>>> 19/404328 objects unfound (0.005%) >>>>>>>>>>>> 2020-10-26T12:54:22.541-0400 7f2a8112b700 0 >>>>>>>>>>>> log_channel(cluster) do_log log to syslog >>>>>>>>>>>> --------------- >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>> >>>>>>>>>> Ok i will do that... but the thing is that the cluster not >>>>>>>>>> show recovering, not show that are doing nothing, like to >>>>>>>>>> show the recovering info on ceph -s command, and then i >>>>>>>>>> dont know if is recovering or doing what?

1 0

OSD utilization vs PG shard sum
by Jonas Jelten 27 Oct '20

27 Oct '20

Hi! I'm creating a custom balancer (mail will follow soonish), and for it to work, I'm calculating OSD usages manually, with the goal to simulate PG movements. But the calculations don't match up, I'm missing some size component. So far I have: osd_used_bytes = sum(pg_shardsizes) But that does not add up: used = ceph osd df -> osd['kb_used'] occupied = pg shard sum 404 used=1.080T occupied=1.137T => 58.303G 405 used=1.031T occupied=1.089T => 59.255G 406 used=4.459T occupied=4.563T => 105.685G 407 used=4.414T occupied=4.433T => 19.751G 408 used=4.428T occupied=4.449T => 21.906G 409 used=4.440T occupied=4.417T => -23.441G 410 used=4.416T occupied=4.397T => -19.868G 411 used=4.446T occupied=4.488T => 42.905G 412 used=4.414T occupied=4.386T => -28.452G 413 used=4.439T occupied=4.461T => 23.326G Especially weird are the negative deltas, they mean the sum of pg shardsizes is more than the osd-reported size. Could this be compression? If yes, how can I get per-PG compression stats? Missing from the calculation is the bluefs_db_size (is that available via json? I could only find the daemon-socket perf counters and prometheus). It's around 2G for each OSD, and should not contribute much to the delta. In short: How do I reliably calculate the real OSD utilization when summing up sizes of currently mapped PGs? -- Jonas

1 0

Strange USED size
by Marcelo 27 Oct '20

27 Oct '20

Hello. I've searched a lot but couldn't find why the size of USED column in the output of ceph df is a lot times bigger than the actual size. I'm using Nautilus (14.2.8), and I've 1000 buckets with 100 objectsineach bucket. Each object is around 10B. ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 511 GiB 147 GiB 340 GiB 364 GiB 71.21 TOTAL 511 GiB 147 GiB 340 GiB 364 GiB 71.21 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 1.1 KiB 4 768 KiB 0 36 GiB default.rgw.control 11 0 B 8 0 B 0 36 GiB default.rgw.meta 12 449 KiB 2.00k 376 MiB 0.34 36 GiB default.rgw.log 13 3.4 KiB 207 6 MiB 0 36 GiB default.rgw.buckets.index 14 0 B 1.00k 0 B 0 36 GiB default.rgw.buckets.data 15 969 KiB 100k 18 GiB 14.52 36 GiB default.rgw.buckets.non-ec 16 27 B 1 192 KiB 0 36 GiB Does anyone know what are the maths behind this, to show 18GiB used when I have something like 1 MiB? Thanks in advance, Marcelo.

4 3

Ceph Octopus and Snapshot Schedules
by Adam Boyhan 27 Oct '20

27 Oct '20

Hey all. I was wondering if Ceph Octopus is capable of automating/managing snapshot creation/retention and then replication? Ive seen some notes about it, but can't seem to find anything solid. Open to suggestions as well. Appreciate any input!

3 7

OSD disk usage
by Kristof Coucke 27 Oct '20

27 Oct '20

Hi, My cluster is in a warning state as it is rebalancing after I've added a bunch of disks. (no issue here!) Though, there are a few things which I just cannot understand... I hope someone can help me... I'm getting hopeless finding the answers... If you can answer any question (even one), it will be greatly appreciated. Environment: Ceph Nautilus 14.2.11, 282 OSDs, mainly erasure coding. 1. Ceph dashboard shows more PGs for an OSD then I can extract from the pg dump information. I assume that the value in the dashboard is coming from Prometheus, though I'm not sure. Querying Prometheus thru Grafana also gives me the same figures as the one I see in the dashboard (which is incorrect). I can't find out how this value is calculated... All the information I can find regarding calculating the pg's stored on an osd are derived from pg dump :-( Help help help 2. As the cluster is rebalancing and there is a huge gap between the acting and the up osd's for a bunch of pg's, some disks sometimes have slow responses due to the backfilling and are from time to time marked as down. (I know I can go around this one by setting "nodown" temporarily). If the disk which is going down is in an acting set of a specific pg (not in the upmap for that pg), then the pg will be marked as degraded as the system will try to rebuild the missing data towards an osd in the up set. This I understand... What I don't understand is that when the disk is restarted and thus marked back as "up" (or simply being patient...), it doesn't add that osd back to the acting set... Restarting other osds (and thus causing more peering operations again) results in the disk being added again to the acting set... I don't understand how this is happening. 3. Addition to question 2: If an osd which was in an acting set went down and is removed from the acting set (replaced with -1), which process will remove the obsolete data from the osd that went down once it is back up? Which process is cleaning up the obsolete copy in the end? Does scrubbing take this process into account? I assume that this might be related to my first question too. 4. I'm data mining the pg dump output... If I get an answer on all the previous questions, this will probably be answered automatically: When I look at all the acting pg's for a specific osd, and I look at the numbytes of that pg and I calculate the size that should be stored on that osd (taking into account the erasure coding process) I get a difference between disk space which should be used and effectively used. E.g. for a disk the system is saying 11.3 TiB is being used... Calculating it using pg dump gives me +/- 10TiB which should be used. I know that a delta might occur due to the block size etc, but it doesn't seem correct as the usage is too high... I've tried to search on processes which clean up the osds, garbage collection, etc. but no good information is available. You can find tons of information related to garbage collection in combination with RGW, but not for the RADOS mechanism... I really can't find any clue how the pg's of a disk are removed after it went down and is not being used again in the acting/up set of those pg's... Another question which I probably should post in the dev group, but which IDE is recommended for developing in the Ceph project? I'm working on a Mac... Don't know if there are any recommendations. I really hope I can get some help on these questions. Many thanks! Regards, Kristof

2 1

Re: Ceph Octopus and Snapshot Schedules
by Adam Boyhan 27 Oct '20

27 Oct '20

Benji is independent from Ceph. It utilizes Ceph snapshots to do the backups, but it has nothing to do with managing Ceph snapshots. I am simply looking for the ability to manage Ceph snapshots. For example. Take a snapshot every 30 minutes, keep 8 of those 30 minute snapshots.

1 0

Switching to a private repository
by Liam MacKenzie 26 Oct '20

26 Oct '20

Hi, What is the correct procedure to change a running cluster deployed with cephadm to a custom container from a private repository that requires a login? I would have thought something like this would have been the right way, but the second command fails with an authentication error. $ cephadm registry-login --registry-url myregistry.com/ceph/ --registry-username 'robot$myuser --registry-password 'mysecret' $ ceph orch upgrade start --image myregistry.com/ceph/ceph:v15.2.5 Thanks, Liam

2 1

Cephadm: module not found
by Marco Venuti 26 Oct '20

26 Oct '20

Hi, I'm experimenting ceph on a (small) test cluster. I'm using version 15.2.5 deployed with cephadm. I was trying to do some "disaster" testing, such as wiping a disk in order to simulate a hardware failure, destroy the osd and recreate it, all of which I managed to do successfully. However, a few hours after this test, the orchestrator failed with no apparent reason. I tried to disable and reenable cephadm, but with no luck: # ceph orch ls Error ENOENT: No orchestrator configured (try `ceph orch set backend`) # ceph orch set backend cephadm Error ENOENT: Module not found What could have happened? Is there some way to reenable cephadm? Thanks, Marco

3 6

Ceph not showing full capacity
by Amudhan P 26 Oct '20

26 Oct '20

Hi, I have created a test Ceph cluster with Ceph Octopus using cephadm. Cluster total RAW disk capacity is 262 TB but it's allowing to use of only 132TB. I have not set quota for any of the pool. what could be the issue? Output from :- ceph -s cluster: id: f8bc7682-0d11-11eb-a332-0cc47a5ec98a health: HEALTH_WARN clock skew detected on mon.strg-node3, mon.strg-node2 2 backfillfull osd(s) 4 pool(s) backfillfull 1 pools have too few placement groups services: mon: 3 daemons, quorum strg-node1,strg-node3,strg-node2 (age 7m) mgr: strg-node3.jtacbn(active, since 7m), standbys: strg-node1.gtlvyv mds: cephfs-strg:1 {0=cephfs-strg.strg-node1.lhmeea=up:active} 1 up:standby osd: 48 osds: 48 up (since 7m), 48 in (since 5d) task status: scrub status: mds.cephfs-strg.strg-node1.lhmeea: idle data: pools: 4 pools, 289 pgs objects: 17.29M objects, 66 TiB usage: 132 TiB used, 130 TiB / 262 TiB avail pgs: 288 active+clean 1 active+clean+scrubbing+deep mounted volume shows node1:/ 67T 66T 910G 99% /mnt/cephfs

5 13

Re: Huge HDD ceph monitor usage
by Ing. Luis Felipe Domínguez Vega 26 Oct '20

26 Oct '20

Only 9M ------------------------------------------------------------------------------ root@fond-beagle:/var/lib/ceph/mon/ceph-fond-beagle/store.db# ls -lh *.log -rw------- 1 ceph ceph 9.9M Oct 26 10:57 1443554.log ------------------------------------------------------------------------------- El 2020-10-26 10:55, Anthony D'Atri escribió: > See if you have big LOG* files inside misleading you > >> On Oct 26, 2020, at 7:13 AM, Ing. Luis Felipe Domínguez Vega >> <luis.dominguez(a)desoft.cu> wrote: >> >> How can i free the store of ceph monitor?: >> >> ------------------------------------------------------------------------ >> root@fond-beagle:/var/lib/ceph/mon/ceph-fond-beagle# du -h -d1 >> 542G ./store.db >> 542G . >> ------------------------------------------------------------------------ >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Hardware for new OSD nodes.
by Dave Hall 26 Oct '20

26 Oct '20

Hello, (BTW, Nautilus 14.2.7 on Debian non-container.) We're about to purchase more OSD nodes for our cluster, but I have a couple questions about hardware choices. Our original nodes were 8 x 12TB SAS drives and a 1.6TB Samsung NVMe card for WAL, DB, etc. We chose the NVMe card for performance since it has an 8 lane PCIe interface. However, we're currently BlueFS spillovers. The Tyan chassis we are considering has the option of 4 x U.2 NVMe bays - each with 4 PCIe lanes, (and 8 SAS bays). It has occurred to me that I might stripe 4 1TB NVMe drives together to get much more space for WAL/DB and a net performance of 16 PCIe lanes. Any thoughts on this approach? Also, any thoughts/recommendations on 12TB OSD drives? For price/capacity this is a good size for us, but I'm wondering if my BlueFS spillovers are resulting from using drives that are too big. I also thought I might have seen some comments about cutting large drives into multiple OSDs - could that be? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

4 15

Question about expansion existing Ceph cluster - adding OSDs
by Kristof Coucke 26 Oct '20

26 Oct '20

Hi, I have a cluster with 182 OSDs, this has been expanded towards 282 OSDs. Some disks were near full. The new disks have been added with initial weight = 0. The original plan was to increase this slowly towards their full weight using the gentle reweight script. However, this is going way too slow and I'm also having issues now with "backfill_toofull". Can I just add all the OSDs with their full weight, or will I get a lot of issues when I'm doing that? I know that a lot of PGs will have to be replaced, but increasing the weight slowly will take a year at the current speed. I'm already playing with the max backfill to increase the speed, but every time I increase the weight it will take a lot of time again... I can face the fact that there will be a performance decrease. Looking forward to your comments! Regards, Kristof

3 6

Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Jean-Philippe Méthot 26 Oct '20

26 Oct '20

Hi, We’re upgrading our cluster OSD node per OSD node to Nautilus from Mimic. From some release notes, it was recommended to run the following command to fix stats after an upgrade : ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 However, running that command gives us the following error message: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: In > function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: 53 > : FAILED ceph_assert(r == 0) > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f1a5a823025] > 2: (()+0x25c1ed) [0x7f1a5a8231ed] > 3: (()+0x3c7a4f) [0x55b33537ca4f] > 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 8: (main()+0x10b3) [0x55b335187493] > 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 10: (()+0x1f9b5f) [0x55b3351aeb5f] > 2020-09-10 14:40:25.873 7f1a6467eec0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc: 53: FAILED ceph_assert(r == 0) > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f1a5a823025] > 2: (()+0x25c1ed) [0x7f1a5a8231ed] > 3: (()+0x3c7a4f) [0x55b33537ca4f] > 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 8: (main()+0x10b3) [0x55b335187493] > 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 10: (()+0x1f9b5f) [0x55b3351aeb5f] > *** Caught signal (Aborted) ** > in thread 7f1a6467eec0 thread_name:ceph-bluestore- > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (()+0xf630) [0x7f1a58cf0630] > 2: (gsignal()+0x37) [0x7f1a574be387] > 3: (abort()+0x148) [0x7f1a574bfa78] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f1a5a823074] > 5: (()+0x25c1ed) [0x7f1a5a8231ed] > 6: (()+0x3c7a4f) [0x55b33537ca4f] > 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 11: (main()+0x10b3) [0x55b335187493] > 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 13: (()+0x1f9b5f) [0x55b3351aeb5f] > 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal (Aborted) ** > in thread 7f1a6467eec0 thread_name:ceph-bluestore- > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (()+0xf630) [0x7f1a58cf0630] > 2: (gsignal()+0x37) [0x7f1a574be387] > 3: (abort()+0x148) [0x7f1a574bfa78] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f1a5a823074] > 5: (()+0x25c1ed) [0x7f1a5a8231ed] > 6: (()+0x3c7a4f) [0x55b33537ca4f] > 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] > 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] > 9: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x55b335274528] > 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) [0x55b3352749a1] > 11: (main()+0x10b3) [0x55b335187493] > 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] > 13: (()+0x1f9b5f) [0x55b3351aeb5f] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. What could be the source of this error? I haven’t found much of anything about it online. Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. 4414-4416 Louis B Mayer Laval, QC, H7P 0G1, Canada TEL : +1.514.802.1644 - Poste : 2644 FAX : +1.514.612.0678 CA/US : 1.855.774.4678 FR : 01 76 60 41 43 UK : 0808 189 0423

3 7

Re: Issues with the ceph-bluestore-tool during cluster upgrade from Mimic to Nautilus
by Igor Fedotov 26 Oct '20

26 Oct '20

On 9/25/2020 6:07 PM, Saber(a)PlanetHoster.info wrote: > Hi Igor, > > The only thing abnormal about this osdstore is that it was created by > Mimic 13.2.8 and I can see that the OSDs size of this osdstore are not > the same as the others in the cluster (while they should be exactly > the same size). > > Can it be https://tracker.ceph.com/issues/39151 ? hmm, may be... Did you change H/W at some point for this OSD's node as it happened in the ticket? And it's still unclear to me if the issue is reproducible for you. Could you please also run fsck (at first) and then repair for this OSD and collect log(s). Thanks, Igor > > Thanks! > Saber > CTO @PlanetHoster > >> On Sep 25, 2020, at 5:46 AM, Igor Fedotov <ifedotov(a)suse.de >> <mailto:ifedotov@suse.de>> wrote: >> >> Hi Saber, >> >> I don't think this is related. New assertion happens along the write >> path while the original one occurred on allocator shutdown. >> >> >> Unfortunately there are not much information to troubleshoot this... >> Are you able to reproduce the case? >> >> >> Thanks, >> >> Igor >> >> On 9/25/2020 4:21 AM, Saber(a)PlanetHoster.info wrote: >>> Hi Igor, >>> >>> We had an osd crash a week after running Nautilus. I have attached >>> the logs, is it related to the same bug? >>> >>> >>> >>> >>> Thanks, >>> Saber >>> CTO @PlanetHoster >>> >>>> On Sep 14, 2020, at 10:22 AM, Igor Fedotov <ifedotov(a)suse.de >>>> <mailto:ifedotov@suse.de>> wrote: >>>> >>>> Thanks! >>>> >>>> Now got the root cause. The fix is on its way... >>>> >>>> Meanwhile you might want to try to workaround the issue via setting >>>> "bluestore_hybrid_alloc_mem_cap" to 0 or using different allocator, >>>> e.g. avl for bluestore_allocator (and optionally for >>>> bluefs_allocator too). >>>> >>>> >>>> Hope this helps, >>>> >>>> Igor. >>>> >>>> >>>> >>>> On 9/14/2020 5:02 PM, Jean-Philippe Méthot wrote: >>>>> Alright, here’s the full log file. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Jean-Philippe Méthot >>>>> Senior Openstack system administrator >>>>> Administrateur système Openstack sénior >>>>> PlanetHoster inc. >>>>> 4414-4416 Louis B Mayer >>>>> Laval, QC, H7P 0G1, Canada >>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>> FAX : +1.514.612.0678 >>>>> CA/US : 1.855.774.4678 >>>>> FR : 01 76 60 41 43 >>>>> UK : 0808 189 0423 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Le 14 sept. 2020 à 06:49, Igor Fedotov <ifedotov(a)suse.de >>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>> >>>>>> Well, I can see duplicate admin socket command >>>>>> registration/de-registration (and the second de-registration >>>>>> asserts) but don't understand how this could happen. >>>>>> >>>>>> Would you share the full log, please? >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Igor >>>>>> >>>>>> On 9/11/2020 7:26 PM, Jean-Philippe Méthot wrote: >>>>>>> Here’s the out file, as requested. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Jean-Philippe Méthot >>>>>>> Senior Openstack system administrator >>>>>>> Administrateur système Openstack sénior >>>>>>> PlanetHoster inc. >>>>>>> 4414-4416 Louis B Mayer >>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>> FAX : +1.514.612.0678 >>>>>>> CA/US : 1.855.774.4678 >>>>>>> FR : 01 76 60 41 43 >>>>>>> UK : 0808 189 0423 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Le 11 sept. 2020 à 10:38, Igor Fedotov <ifedotov(a)suse.de >>>>>>>> <mailto:ifedotov@suse.de>> a écrit : >>>>>>>> >>>>>>>> Could you please run: >>>>>>>> >>>>>>>> CEPH_ARGS="--log-file log --debug-asok 5" ceph-bluestore-tool >>>>>>>> repair --path <...> ; cat log | grep asok > out >>>>>>>> >>>>>>>> and share 'out' file. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Igor >>>>>>>> >>>>>>>> On 9/11/2020 5:15 PM, Jean-Philippe Méthot wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> We’re upgrading our cluster OSD node per OSD node to Nautilus >>>>>>>>> from Mimic. From some release notes, it was recommended to run >>>>>>>>> the following command to fix stats after an upgrade : >>>>>>>>> >>>>>>>>> ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0 >>>>>>>>> >>>>>>>>> However, running that command gives us the following error >>>>>>>>> message: >>>>>>>>> >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: In >>>>>>>>>> function 'virtual Allocator::SocketHook::~SocketHook()' >>>>>>>>>> thread 7f1a6467eec0 time 2020-09-10 14:40:25.872353 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: 53 >>>>>>>>>> : FAILED ceph_assert(r == 0) >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> 2020-09-10 14:40:25.873 7f1a6467eec0 -1 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: In function 'virtual >>>>>>>>>> Allocator::SocketHook::~SocketHook()' thread 7f1a6467eec0 >>>>>>>>>> time 2020-09-10 14:40:25.872353 >>>>>>>>>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/Allocator.cc >>>>>>>>>> <http://allocator.cc/>: 53: FAILED ceph_assert(r == 0) >>>>>>>>>> >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x14a) [0x7f1a5a823025] >>>>>>>>>> 2: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 3: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 4: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 5: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 6: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 8: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 9: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 10: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> *** Caught signal (Aborted) ** >>>>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> 2020-09-10 14:40:25.874 7f1a6467eec0 -1 *** Caught signal >>>>>>>>>> (Aborted) ** >>>>>>>>>> in thread 7f1a6467eec0 thread_name:ceph-bluestore- >>>>>>>>>> >>>>>>>>>> ceph version 14.2.11 >>>>>>>>>> (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) >>>>>>>>>> 1: (()+0xf630) [0x7f1a58cf0630] >>>>>>>>>> 2: (gsignal()+0x37) [0x7f1a574be387] >>>>>>>>>> 3: (abort()+0x148) [0x7f1a574bfa78] >>>>>>>>>> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, >>>>>>>>>> char const*)+0x199) [0x7f1a5a823074] >>>>>>>>>> 5: (()+0x25c1ed) [0x7f1a5a8231ed] >>>>>>>>>> 6: (()+0x3c7a4f) [0x55b33537ca4f] >>>>>>>>>> 7: (HybridAllocator::~HybridAllocator()+0x17) [0x55b3353ac517] >>>>>>>>>> 8: (BlueStore::_close_alloc()+0x42) [0x55b3351f2082] >>>>>>>>>> 9: (BlueStore::_close_db_and_around(bool)+0x2f8) >>>>>>>>>> [0x55b335274528] >>>>>>>>>> 10: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x2c1) >>>>>>>>>> [0x55b3352749a1] >>>>>>>>>> 11: (main()+0x10b3) [0x55b335187493] >>>>>>>>>> 12: (__libc_start_main()+0xf5) [0x7f1a574aa555] >>>>>>>>>> 13: (()+0x1f9b5f) [0x55b3351aeb5f] >>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS >>>>>>>>>> <executable>` is needed to interpret this. >>>>>>>>> >>>>>>>>> What could be the source of this error? I haven’t found much >>>>>>>>> of anything about it online. >>>>>>>>> >>>>>>>>> >>>>>>>>> Jean-Philippe Méthot >>>>>>>>> Senior Openstack system administrator >>>>>>>>> Administrateur système Openstack sénior >>>>>>>>> PlanetHoster inc. >>>>>>>>> 4414-4416 Louis B Mayer >>>>>>>>> Laval, QC, H7P 0G1, Canada >>>>>>>>> TEL : +1.514.802.1644 - Poste : 2644 >>>>>>>>> FAX : +1.514.612.0678 >>>>>>>>> CA/US : 1.855.774.4678 >>>>>>>>> FR : 01 76 60 41 43 >>>>>>>>> UK : 0808 189 0423 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> <mailto:ceph-users@ceph.io> >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> <mailto:ceph-users-leave@ceph.io> >>>>>>> >>>>> >>> >

2 1

Ceph docker containers all stopped
by Darrin Hodges 26 Oct '20

26 Oct '20

HI all, Had an issue where the docker containers on all the ceph nodes just seem to stop at some point, effectively shutting down the cluster. Restarting cephs on all of the nodes restored the cluster to normal working order. I would like to find out why this occurred, any ideas on where to look? many thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

2 1

Re: The feasibility of mixed SSD and HDD replicated pool
by 胡玮文 25 Oct '20

25 Oct '20

> 在 2020年10月26日，00:07，Anthony D'Atri <anthony.datri(a)gmail.com> 写道： > >> I'm not entirely sure if primary on SSD will actually make the read happen on SSD. > > My understanding is that by default reads always happen from the lead OSD in the acting set. Octopus seems to (finally) have an option to spread the reads around, which IIRC defaults to false. I also remember that “by default reads always happen from the lead OSD in the acting set”. I dig through git blame and it seems ceph-fuse has a —localize-reads options since 10 years ago [1], through not documented anywhere. I don’t find such setting in kernel ceph module. [1]: https://github.com/ceph/ceph/commit/7912f5c7034bd26d22615d1be1d398849e124749 > I’ve never seen anything that implies that lead OSDs within an acting set are a function of CRUSH rule ordering. I’m not asserting that they aren’t though, but I’m … skeptical. That conclusion is from experiments. I create an empty pool with above mentioned CRUSH rule, and all 32 pgs have SSD as primary. > Setting primary affinity would do the job, and you’d want to have cron continually update it across the cluster to react to topology changes. I was told of this strategy back in 2014, but haven’t personally seen it implemented. I’m also considering this. But if I set the primary affinity of HDDs to 0, then what will happen if I create another all-HDD pool? Or I should just set primary affinity to a very small value, say 0.00001. > That said, HDDs are more of a bottleneck for writes than reads and just might be fine for your application. Tiny reads are going to limit you to some degree regardless of drive type, and you do mention throughput, not IOPS. > > I must echo Frank’s notes about capacity too. Ceph can do a lot of things, but that doesn’t mean something exotic is necessarily the best choice. You’re concerned about 3R only yielding 1/3 of raw capacity if using an all-SSD cluster, but the architecture you propose limits you anyway because drive size. Consider also chassis, CPU, RAM, RU, switch port costs as well, and the cost of you fussing over an exotic solution instead of the hundreds of other things in your backlog. > > And your cluster as described is *tiny*. Honestly I’d suggest considering one of these alternatives: > > * Ditch the HDDs, use QLC flash. The emerging EDSFF drives are really promising for replacing HDDs for density in this kind of application. You might even consider ARM if IOPs aren’t a concern. > * An NVMeoF solution Thanks for the advices, we will discuss on these. But this deployment is on existing server hardwares, so we don’t have many choices. And our budget is very limited. We want to make best use of our existing SSDs. And we have plenty of cold data to fill our HDDs. We will not worry about the wasting of HDD capacity. Sorry Anthony, I sent this mail twice. I forgot to CC this mail list at first. > Cache tiers are “deprecated”, but then so are custom cluster names. Neither appears > >> For EC pools there is an option "fast_read" (https://docs.ceph.com/en/latest/rados/operations/pools/?highlight=fast_read…) which states that a read will return as soon as the first k shards have arrived. The default is to wait for all k+m shards (all replicas). This option is not available for replicated pools. >> >> Now, not sure if this option is not available for replicated pools because the read will always be served by the acting primary, or if it currently waits for all replicas. In the latter case, reads will wait for the slowest device. >> >> I'm not sure if I interpret this correctly. I think you should test the setup with HDD only and SSD+HDD to see if read speed improves. Note that write speed will always depend on the slowest device. >> >> Best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> >> ________________________________________ >> From: Frank Schilder <frans(a)dtu.dk> >> Sent: 25 October 2020 15:03:16 >> To: 胡玮文; Alexander E. Patrakov >> Cc: ceph-users(a)ceph.io >> Subject: [ceph-users] Re: The feasibility of mixed SSD and HDD replicated pool >> >> A cache pool might be an alternative, heavily depending on how much data is hot. However, then you will have much less SSD capacity available, because it also requires replication. >> >> Looking at the setup that you have only 10*1T =10T SSD, but 20*6T = 120T HDD you will probably run short of SSD capacity. Or, looking at it the other way around, with copies on 1 SSD+3HDD, you will only be able to use about 30T out of 120T HDD capacity. >> >> With this replication, the usable storage will be 10T and raw used will be 10T SSD and 30T HDD. If you can't do anything else on the HDD space, you will need more SSDs. If your servers have more free disk slots, you can add SSDs over time until you have at least 40T SSD capacity to balance SSD and HDD capacity. >> >> Personally, I think the 1SSD + 3HDD is a good option compared with a cache pool. You have the data security of 3-times replication and, if everything is up, need only 1 copy in the SSD cache, which means that you have 3 times the cache capacity. >> >> Best regards, >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> >> ________________________________________ >> From: 胡玮文 <huww98(a)outlook.com> >> Sent: 25 October 2020 13:40:55 >> To: Alexander E. Patrakov >> Cc: ceph-users(a)ceph.io >> Subject: [ceph-users] Re: The feasibility of mixed SSD and HDD replicated pool >> >> Yes. This is the limitation of CRUSH algorithm, in my mind. In order to guard against 2 host failures, I’m going to use 4 replications, 1 on SSD and 3 on HDD. This will work as intended, right? Because at least I can ensure 3 HDDs are from different hosts. >> >>>> 在 2020年10月25日，20:04，Alexander E. Patrakov <patrakov(a)gmail.com> 写道： >>> >>> On Sun, Oct 25, 2020 at 12:11 PM huww98(a)outlook.com <huww98(a)outlook.com> wrote: >>>> >>>> Hi all, >>>> >>>> We are planning for a new pool to store our dataset using CephFS. These data are almost read-only (but not guaranteed) and consist of a lot of small files. Each node in our cluster has 1 * 1T SSD and 2 * 6T HDD, and we will deploy about 10 such nodes. We aim at getting the highest read throughput. >>>> >>>> If we just use a replicated pool of size 3 on SSD, we should get the best performance, however, that only leave us 1/3 of usable SSD space. And EC pools are not friendly to such small object read workload, I think. >>>> >>>> Now I’m evaluating a mixed SSD and HDD replication strategy. Ideally, I want 3 data replications, each on a different host (fail domain). 1 of them on SSD, the other 2 on HDD. And normally every read request is directed to SSD. So, if every SSD OSD is up, I’d expect the same read throughout as the all SSD deployment. >>>> >>>> I’ve read the documents and did some tests. Here is the crush rule I’m testing with: >>>> >>>> rule mixed_replicated_rule { >>>> id 3 >>>> type replicated >>>> min_size 1 >>>> max_size 10 >>>> step take default class ssd >>>> step chooseleaf firstn 1 type host >>>> step emit >>>> step take default class hdd >>>> step chooseleaf firstn -1 type host >>>> step emit >>>> } >>>> >>>> Now I have the following conclusions, but I’m not very sure: >>>> * The first OSD produced by crush will be the primary OSD (at least if I don’t change the “primary affinity”). So, the above rule is guaranteed to map SSD OSD as primary in pg. And every read request will read from SSD if it is up. >>>> * It is currently not possible to enforce SSD and HDD OSD to be chosen from different hosts. So, if I want to ensure data availability even if 2 hosts fail, I need to choose 1 SSD and 3 HDD OSD. That means setting the replication size to 4, instead of the ideal value 3, on the pool using the above crush rule. >>>> >>>> Am I correct about the above statements? How would this work from your experience? Thanks. >>> >>> This works (i.e. guards against host failures) only if you have >>> strictly separate sets of hosts that have SSDs and that have HDDs. >>> I.e., there should be no host that has both, otherwise there is a >>> chance that one hdd and one ssd from that host will be picked. >>> >>> -- >>> Alexander E. Patrakov >>> CV: https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpc.cd%2FPL… >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph cluster recovering status
by Ing. Luis Felipe Domínguez Vega 25 Oct '20

25 Oct '20

Hi, my cluster was crashed by going down one of my DC and 'ceph -s' status dont show me the current working status and nothing change in large time, how can i see what is ceph doing really: cluster: health: HEALTH_ERR mons fond-beagle,guided-tuna are using a lot of disk space 1/3 mons down, quorum fond-beagle,guided-tuna 18/404368 objects unfound (0.004%) Reduced data availability: 235 pgs inactive, 72 pgs down, 9 pgs incomplete Possible data damage: 3 pgs recovery_unfound Degraded data redundancy: 306574/2607020 objects degraded (11.760%), 10 pgs degraded, 10 pgs undersized 2 pgs not deep-scrubbed in time 32408 slow ops, oldest one blocked for 62348 sec, daemons [osd.0,osd.10,osd.11,osd.13,osd.14,osd.15,osd.16,osd.17,osd.18,osd.19]... have slow ops. services: mon: 3 daemons, quorum fond-beagle,guided-tuna (age 31m), out of quorum: alive-lynx mgr: fond-beagle(active, since 31m) osd: 52 osds: 28 up (since 30m), 28 in (since 11h); 3 remapped pgs data: pools: 7 pools, 2305 pgs objects: 404.37k objects, 1.7 TiB usage: 2.7 TiB used, 22 TiB / 24 TiB avail pgs: 6.681% pgs unknown 3.514% pgs not active 306574/2607020 objects degraded (11.760%) 18/404368 objects unfound (0.004%) 2060 active+clean 154 unknown 72 down 9 incomplete 7 active+undersized+degraded 3 active+recovery_unfound+undersized+degraded+remapped

1 0

Urgent help needed please - MDS offline
by David C 23 Oct '20

23 Oct '20

Hi All My main CephFS data pool on a Luminous 12.2.10 cluster hit capacity overnight, metadata is on a separate pool which didn't hit capacity but the filesystem stopped working which I'd expect. I increased the osd full-ratio to give me some breathing room to get some data deleted once the filesystem is back online. When I attempt to restart the MDS service, I see the usual stuff I'd expect in the log but then: heartbeat_map is_healthy 'MDSRank' had timed out after 15 Followed by: mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last > acked 4.00013s ago); MDS internal heartbeat is not healthy! Eventually I get: > > mds.beacon.hostnamecephssd01 is_laggy 29.372 > 15 since last acked beacon > mds.0.90884 skipping upkeep work because connection to Monitors appears > laggy > mds.hostnamecephssd01 Updating MDS map to version 90885 from mon.0 > mds.beacon.hostnamecephssd01 MDS is no longer laggy The "MDS is no longer laggy" appears to be where the service fails Meanwhile a ceph -s is showing: > > cluster: > id: 5c5998fd-dc9b-47ec-825e-beaba66aad11 > health: HEALTH_ERR > 1 filesystem is degraded > insufficient standby MDS daemons available > 67 backfillfull osd(s) > 11 nearfull osd(s) > full ratio(s) out of order > 2 pool(s) backfillfull > 2 pool(s) nearfull > 6 scrub errors > Possible data damage: 5 pgs inconsistent > services: > mon: 3 daemons, quorum hostnameceph01,hostnameceph02,hostnameceph03 > mgr: hostnameceph03(active), standbys: hostnameceph02, hostnameceph01 > mds: cephfs-1/1/1 up {0=hostnamecephssd01=up:replay} > osd: 172 osds: 161 up, 161 in > data: > pools: 5 pools, 8384 pgs > objects: 76.25M objects, 124TiB > usage: 373TiB used, 125TiB / 498TiB avail > pgs: 8379 active+clean > 5 active+clean+inconsistent > io: > client: 676KiB/s rd, 0op/s rd, 0op/s w The 5 pgs inconsistent is not a new issue, that is from past scrubs, just haven't gotten around to manually clearing them although I suppose they could be related to my issue The cluster has no clients connected I did notice in the ceph.log, some OSDs that are in the same host as the MDS service briefly went down when trying to restart the MDS but examining the logs of those particular OSDs isn't showing any glaring issues. Full MDS log at debug 5 (can go higher if needed): 2020-10-22 11:27:10.987652 7f6f696f5240 0 set uid:gid to 167:167 (ceph:ceph) 2020-10-22 11:27:10.987669 7f6f696f5240 0 ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) luminous (stable), process ceph-mds, pid 2022582 2020-10-22 11:27:10.990567 7f6f696f5240 0 pidfile_write: ignore empty --pid-file 2020-10-22 11:27:11.027981 7f6f62616700 1 mds.hostnamecephssd01 Updating MDS map to version 90882 from mon.0 2020-10-22 11:27:15.097957 7f6f62616700 1 mds.hostnamecephssd01 Updating MDS map to version 90883 from mon.0 2020-10-22 11:27:15.097989 7f6f62616700 1 mds.hostnamecephssd01 Map has assigned me to become a standby 2020-10-22 11:27:15.101071 7f6f62616700 1 mds.hostnamecephssd01 Updating MDS map to version 90884 from mon.0 2020-10-22 11:27:15.105310 7f6f62616700 1 mds.0.90884 handle_mds_map i am now mds.0.90884 2020-10-22 11:27:15.105316 7f6f62616700 1 mds.0.90884 handle_mds_map state change up:boot --> up:replay 2020-10-22 11:27:15.105325 7f6f62616700 1 mds.0.90884 replay_start 2020-10-22 11:27:15.105333 7f6f62616700 1 mds.0.90884 recovery set is 2020-10-22 11:27:15.105344 7f6f62616700 1 mds.0.90884 waiting for osdmap 73745 (which blacklists prior instance) 2020-10-22 11:27:15.149092 7f6f5be09700 0 mds.0.cache creating system inode with ino:0x100 2020-10-22 11:27:15.149693 7f6f5be09700 0 mds.0.cache creating system inode with ino:0x1 2020-10-22 11:27:41.021708 7f6f63618700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:27:43.029290 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:27:43.029297 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 4.00013s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:27:45.866711 7f6f5fe11700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-10-22 11:28:01.021965 7f6f63618700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:03.029862 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:03.029885 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 4.00113s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:28:06.022033 7f6f63618700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:07.029955 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:07.029961 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 8.00126s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:28:11.022099 7f6f63618700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:11.030024 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:11.030028 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 12.0014s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:28:15.030092 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:15.030099 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 16.0015s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:28:16.022165 7f6f63618700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:19.030163 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:19.030169 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 20.0016s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:28:21.022231 7f6f63618700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:23.030233 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:23.030241 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 24.0008s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:28:26.022295 7f6f63618700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:27.030305 7f6f5f610700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:28:27.030311 7f6f5f610700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 28.0009s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:28:28.401161 7f6f5fe11700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-10-22 11:28:28.401168 7f6f5fe11700 1 mds.beacon.hostnamecephssd01 is_laggy 29.372 > 15 since last acked beacon 2020-10-22 11:28:28.401177 7f6f5fe11700 1 mds.0.90884 skipping upkeep work because connection to Monitors appears laggy 2020-10-22 11:28:28.401187 7f6f62616700 1 mds.hostnamecephssd01 Updating MDS map to version 90885 from mon.0 2020-10-22 11:28:31.659817 7f6f64595700 0 mds.beacon.hostnamecephssd01 MDS is no longer laggy 2020-10-22 11:36:15.880009 7f88ee4ac240 0 set uid:gid to 167:167 (ceph:ceph) 2020-10-22 11:36:15.880026 7f88ee4ac240 0 ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) luminous (stable), process ceph-mds, pid 2022663 2020-10-22 11:36:15.883118 7f88ee4ac240 0 pidfile_write: ignore empty --pid-file 2020-10-22 11:36:15.921200 7f88e73cd700 1 mds.hostnamecephssd01 Updating MDS map to version 90887 from mon.2 2020-10-22 11:36:20.270298 7f88e73cd700 1 mds.hostnamecephssd01 Updating MDS map to version 90888 from mon.2 2020-10-22 11:36:20.270329 7f88e73cd700 1 mds.hostnamecephssd01 Map has assigned me to become a standby 2020-10-22 11:36:20.272917 7f88e73cd700 1 mds.hostnamecephssd01 Updating MDS map to version 90889 from mon.2 2020-10-22 11:36:20.277063 7f88e73cd700 1 mds.0.90889 handle_mds_map i am now mds.0.90889 2020-10-22 11:36:20.277069 7f88e73cd700 1 mds.0.90889 handle_mds_map state change up:boot --> up:replay 2020-10-22 11:36:20.277079 7f88e73cd700 1 mds.0.90889 replay_start 2020-10-22 11:36:20.277086 7f88e73cd700 1 mds.0.90889 recovery set is 2020-10-22 11:36:20.277096 7f88e73cd700 1 mds.0.90889 waiting for osdmap 73746 (which blacklists prior instance) 2020-10-22 11:36:20.322318 7f88e0bc0700 0 mds.0.cache creating system inode with ino:0x100 2020-10-22 11:36:20.322918 7f88e0bc0700 0 mds.0.cache creating system inode with ino:0x1 2020-10-22 11:36:47.922531 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:36:47.922549 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 4.00013s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:36:50.914516 7f88e83cf700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:36:51.351457 7f88e4bc8700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-10-22 11:37:07.923089 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:07.923126 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 3.99913s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:37:10.914767 7f88e83cf700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:11.923216 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:11.923223 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 7.99926s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:37:15.914831 7f88e83cf700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:15.923286 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:15.923294 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 11.9994s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:37:19.923359 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:19.923366 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 15.9995s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:37:20.914917 7f88e83cf700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:23.923430 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:23.923437 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 19.9996s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:37:25.914981 7f88e83cf700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:27.923501 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:27.923508 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 23.9998s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:37:30.915046 7f88e83cf700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:31.923572 7f88e43c7700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 2020-10-22 11:37:31.923579 7f88e43c7700 0 mds.beacon.hostnamecephssd01 Skipping beacon heartbeat to monitors (last acked 27.9999s ago); MDS internal heartbeat is not healthy! 2020-10-22 11:37:32.412628 7f88e4bc8700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 2020-10-22 11:37:32.412635 7f88e4bc8700 1 mds.beacon.hostnamecephssd01 is_laggy 28.4889 > 15 since last acked beacon 2020-10-22 11:37:32.412643 7f88e4bc8700 1 mds.0.90889 skipping upkeep work because connection to Monitors appears laggy 2020-10-22 11:37:32.412657 7f88e73cd700 1 mds.hostnamecephssd01 Updating MDS map to version 90890 from mon.2 2020-10-22 11:37:35.978858 7f88e934c700 0 mds.beacon.hostnamecephssd01 MDS is no longer laggy Thanks in advance for any assistance you can provide! David

4 15

Ceph and ram limits
by Ing. Luis Felipe Domínguez Vega 23 Oct '20

23 Oct '20

Since some days ago i am recoveryng my ceph cluster, all start with OSD been killed by OOM, well i created a script to delete from the OSD the PGs corrupted (i write corrupted because that pg is the cause of the 100% of RAM usage by OSD). Great, almost done with all OSDs of my cluster, then the monitors now are consuming all the servers RAM, and the Managers too, why??? why they use 60GB of RAM, there are something to block that?? i tried configurind all kind of RAM limit to the minimal.

1 0

Large map object found
by Peter Eisch 23 Oct '20

23 Oct '20

Hi, My rgw.buckets.index has the cluster in WARN. I'm either not understanding the real issue or I'm making it worse, or both. OMAP_BYTES: 70461524 OMAP_KEYS: 250874 I thought I'd head this off by deleting rgw objects which would normally get deleted in the near future but this only seemed to make the values grow. Before I deleted lots of objects the values were: OMAP_BYTES: 65450132 OMAP_KEYS: 209843 I read the default is 200k but I haven't read the proper way to manage this situation. What reading should I dive into? I could probably craft up a command to increase the value to clear the warning but I'm guessing this might not be great long-term. Other errata which might matter: Size: 3 Pool: nvme CLASS SIZE AVAIL USED RAW USED %RAW USED nvme 256 TiB 165 TiB 91 TiB 91 TiB 35.53 Errata: the complete statements: PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 43.d 2 0 0 0 0 70461524 250874 3070 active+clean 36m 185904'456870 185904:1357091 [99,90,48]p99 [99,90,48]p99 2020-10-21 13:53:42.102363 2020-10-21 13:53:42.102363 Thanks! peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.66

2 6

desaster recovery Ceph Storage , urgent help needed
by Gerhard W. Recher 23 Oct '20

23 Oct '20

Hi I have a worst case, osd's in a 3 node cluster each 4 nvme's won't start we had a ip config change in public network, and mon's died so we managed mon's to come back with new ip's. corosync on 2 rings is fine, all 3 mon's are up osd's won't start how to get back to the pool, already 3vm's are configured and valuable data would be lost... this is like a scenario when all systemdisks on each 3 nodes failed, but osd disks are healthy ... any help to reconstruct this storage is highly appreciated! Gerhard |root@pve01:/var/log# systemctl status ceph-osd(a)0.service.service ● ceph-osd(a)0.service.service - Ceph object storage daemon osd.0.service Loaded: loaded (/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: enabled) Drop-In: /usr/lib/systemd/system/ceph-osd@.service.d └─ceph-after-pve-cluster.conf Active: failed (Result: exit-code) since Thu 2020-10-22 00:30:09 CEST; 37min ago Process: 31402 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 0.service (code=exited, status=1/FAILURE) Oct 22 00:30:09 pve01 systemd[1]: ceph-osd(a)0.service.service: Service RestartSec=100ms expired, scheduling restart. Oct 22 00:30:09 pve01 systemd[1]: ceph-osd(a)0.service.service: Scheduled restart job, restart counter is at 3. Oct 22 00:30:09 pve01 systemd[1]: Stopped Ceph object storage daemon osd.0.service. Oct 22 00:30:09 pve01 systemd[1]: ceph-osd(a)0.service.service: Start request repeated too quickly. Oct 22 00:30:09 pve01 systemd[1]: ceph-osd(a)0.service.service: Failed with result 'exit-code'. Oct 22 00:30:09 pve01 systemd[1]: Failed to start Ceph object storage daemon osd.0.service. | ||ceph mon dump dumped monmap epoch 3 epoch 3 fsid 92d063d7-647c-44b8-95d7-86057ee0ab22 last_changed 2020-10-21 23:31:50.584796 created 2020-10-21 21:00:54.077449 min_mon_release 14 (nautilus) 0: [v2:10.100.200.141:3300/0,v1:10.100.200.141:6789/0] mon.pve01 1: [v2:10.100.200.142:3300/0,v1:10.100.200.142:6789/0] mon.pve02 2: [v2:10.100.200.143:3300/0,v1:10.100.200.143:6789/0] mon.pve03 || |||Networks: auto lo iface lo inet loopback auto eno1np0 iface eno1np0 inet static address 10.110.200.131/24 mtu 9000 #corosync1 10GB auto eno2np1 iface eno2np1 inet static address 10.111.200.131/24 mtu 9000 #Corosync2 10GB iface enp69s0f0 inet manual mtu 9000 auto enp69s0f1 iface enp69s0f1 inet static address 10.112.200.131/24 mtu 9000 #Cluster private 100GB auto vmbr0 iface vmbr0 inet static address 10.100.200.141/24 gateway 10.100.200.1 bridge-ports enp69s0f0 bridge-stp off bridge-fd 0 mtu 9000 #Cluster public 100GB =================================================================================================== ||| ||||ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.112.200.0/24 fsid = 92d063d7-647c-44b8-95d7-86057ee0ab22 mon_allow_pool_delete = true mon_host = 10.100.200.141 10.100.200.142 10.100.200.143 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 10.100.200.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mon.pve01] public_addr = 10.100.200.141 [mon.pve02] public_addr = 10.100.200.142 [mon.pve03] public_addr = 10.100.200.143 |||| |||||ceph -s cluster: id: 92d063d7-647c-44b8-95d7-86057ee0ab22 health: HEALTH_WARN 1 daemons have recently crashed OSD count 0 < osd_pool_default_size 3 services: mon: 3 daemons, quorum pve01,pve02,pve03 (age 63m) mgr: pve01(active, since 64m) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: df -h Filesystem Size Used Avail Use% Mounted on udev 252G 0 252G 0% /dev tmpfs 51G 11M 51G 1% /run rpool/ROOT/pve-1 229G 16G 214G 7% / tmpfs 252G 63M 252G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 252G 0 252G 0% /sys/fs/cgroup rpool 214G 128K 214G 1% /rpool rpool/data 214G 128K 214G 1% /rpool/data rpool/ROOT 214G 128K 214G 1% /rpool/ROOT tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-3 tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-2 tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-0 tmpfs 252G 24K 252G 1% /var/lib/ceph/osd/ceph-1 /dev/fuse 30M 32K 30M 1% /etc/pve tmpfs 51G 0 51G 0% /run/user/0 lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme4n1 259:0 0 238.5G 0 disk ├─nvme4n1p1 259:5 0 1007K 0 part ├─nvme4n1p2 259:6 0 512M 0 part └─nvme4n1p3 259:7 0 238G 0 part nvme5n1 259:1 0 238.5G 0 disk ├─nvme5n1p1 259:2 0 1007K 0 part ├─nvme5n1p2 259:3 0 512M 0 part └─nvme5n1p3 259:4 0 238G 0 part nvme0n1 259:12 0 2.9T 0 disk └─ceph--cc77fe1b--c8d4--48be--a7c4--36109439c85c-osd--block--80e0127e--836e--44b8--882d--ac49bfc85866 253:3 0 2.9T 0 lvm nvme1n1 259:13 0 2.9T 0 disk └─ceph--eb8b2fc7--775e--4b94--8070--784e7bbf861e-osd--block--4d433222--e1e8--43ac--8dc7--2e6e998ff122 253:2 0 2.9T 0 lvm nvme3n1 259:14 0 2.9T 0 disk └─ceph--5724bdf7--5124--4244--91d6--e254210c2174-osd--block--2d6fe149--f330--415a--a762--44d037c900b1 253:1 0 2.9T 0 lvm nvme2n1 259:15 0 2.9T 0 disk └─ceph--cb5762e9--40fa--4148--98f4--5b5ddef4c1de-osd--block--793d52e6--4cc7--42aa--8326--df25b21c1237 253:0 0 2.9T 0 lvm |||||

3 5

OSD Failures after pg_num increase on one of the pools
by Артём Григорьев 23 Oct '20

23 Oct '20

Hello everyone, I created a new ceph 14.2.7 Nautilus cluster recently. Cluster consists of 3 racks and 2 osd nodes on each rack, 12 new hdd in each node. HDD model is TOSHIBA MG07ACA14TE 14Tb. All data pools are ec pools. Yesterday I decided to increase pg number on one of the pools with command "ceph osd pool set photo.buckets.data pg_num 512", after that many osds started to crash with "out" and "down" status. I tried to increase recovery_sleep to 1s but osds still crashes. Osds started working properly only when i set "norecover" flag, but osd scrub errors appeared after that. In logs from osd during crashes i found this: --- Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHIN E_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/ECBackend.cc: In function 'void ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)' thread 7f8af535d700 time 2020-10-21 15:12:11.460092 Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHIN E_SIZE/huge/release/14.2.7/rpm/el7/BUILD/ceph-14.2.7/src/osd/ECBackend.cc: 648: FAILED ceph_assert(pop.data.length() == sinfo.aligned_logical_offset_to_chunk_offset( aft er_progress.data_recovered_to - op.recovery_progress.data_recovered_to)) Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55fc694d6c0f] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 2: (()+0x4dddd7) [0x55fc694d6dd7] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 3: (ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)+0x1740) [0x55fc698cafa0] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 4: (ECBackend::handle_recovery_read_complete(hobject_t const&, boost::tuples::tuple<unsigned long, unsigned long, std::map<pg_shard_t, ceph::buffer::v14_2_0::list, std::less<pg_shard_t>, std::allocator<std::pair<pg_shard_t const, ceph::buffer::v14_2_0::list> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>&, boost::optional<std::map<std::string, ceph::buffer::v14_2_0::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::v14_2_0::list> > > >, RecoveryMessages*)+0x734) [0x55fc698cb804] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 5: (OnRecoveryReadComplete::finish(std::pair<RecoveryMessages*, ECBackend::read_result_t&>&)+0x94) [0x55fc698ebbe4] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 6: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x8c) [0x55fc698bfdcc] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 7: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0x109c) [0x55fc698d6b8c] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 8: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x17f) [0x55fc698d718f] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 9: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x4a) [0x55fc697c18ea] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5b3) [0x55fc697676b3] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x362) [0x55fc695b3d72] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 12: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55fc698415c2] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x55fc695cebbf] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55fc69b6f976] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55fc69b72490] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 16: (()+0x7e65) [0x7f8b1ddede65] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: 17: (clone()+0x6d) [0x7f8b1ccb188d] Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: *** Caught signal (Aborted) ** Oct 21 15:12:11 ceph-osd-201 ceph-osd[58159]: in thread 7f8af535d700 thread_name:tp_osd_tp --- Current ec profile and pool info bellow: # ceph osd erasure-code-profile get EC42 crush-device-class=hdd crush-failure-domain=host crush-root=main jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 pool 25 'photo.buckets.data' erasure size 6 min_size 4 crush_rule 6 object_hash rjenkins pg_num 512 pgp_num 280 pgp_num_target 512 autoscale_mode warn last_change 43418 lfor 0/0/42223 flags hashpspool stripe_width 1048576 application rgw Current ceph status: ceph -s cluster: id: 9ec8d309-a620-4ad8-93fa-c2d111e5256e health: HEALTH_ERR norecover flag(s) set 1 pools have many more objects per pg than average 4542629 scrub errors Possible data damage: 6 pgs inconsistent Degraded data redundancy: 1207268/578535561 objects degraded (0.209%), 51 pgs degraded, 35 pgs undersized 85 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph-osd-101,ceph-osd-201,ceph-osd-301 (age 2w) mgr: ceph-osd-101(active, since 3w), standbys: ceph-osd-301, ceph-osd-201 osd: 72 osds: 72 up (since 11h), 72 in (since 21h); 48 remapped pgs flags norecover rgw: 6 daemons active (ceph-osd-101.rgw0, ceph-osd-102.rgw0, ceph-osd-201.rgw0, ceph-osd-202.rgw0, ceph-osd-301.rgw0, ceph-osd-302.rgw0) data: pools: 26 pools, 15680 pgs objects: 96.46M objects, 124 TiB usage: 303 TiB used, 613 TiB / 917 TiB avail pgs: 1207268/578535561 objects degraded (0.209%) 14068769/578535561 objects misplaced (2.432%) 15290 active+clean 312 active+recovering 30 active+undersized+degraded+remapped+backfilling 21 active+recovering+degraded 13 active+remapped+backfilling 6 active+clean+inconsistent 5 active+recovering+undersized+remapped 3 active+clean+scrubbing+deep So now my cluster is stuck and can't recover properly, can someone give info about this problem? Is it a bug?

3 2

Ceph Octopus
by Amudhan P 23 Oct '20

23 Oct '20

Hi, I have installed Ceph Octopus cluster using cephadm with a single network now I want to add a second network and configure it as a cluster address. How do I configure ceph to use second Network as cluster network?. Amudhan

3 6

TOO_FEW_PGS warning and pg_autoscale
by Peter Eisch 23 Oct '20

23 Oct '20

Hi, # ceph health detail HEALTH_WARN too few PGs per OSD (24 < min 30) TOO_FEW_PGS too few PGs per OSD (24 < min 30) ceph version 14.2.9 This warning popped up when autoscale shrunk a pool from pg_num and pgp_num from 512 to 256 on its own. The hdd35 storage is only used by this pool. I have three different storage classes and the pools use the different classes as appropriate. How can I convert the warning into something useful which then helps me make the appropriate change to the right class of storage? I'm guessing it's referring to hdd35. RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd25 129 TiB 83 TiB 46 TiB 46 TiB 35.87 hdd35 269 TiB 220 TiB 49 TiB 49 TiB 18.12 ssd 256 TiB 164 TiB 92 TiB 92 TiB 35.84 TOTAL 655 TiB 468 TiB 186 TiB 187 TiB 28.56 If I follow: https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-few-pgs Which then links to: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#choosing… The math for this would want the pool to have pg/p_num of 2048 -- where autoscale just recently shrunk the count. Which is more right? Thanks! peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.66

1 0

Hardware needs for MDS for HPC/OpenStack workloads?
by Matthew Vernon 23 Oct '20

23 Oct '20

Hi, We're considering the merits of enabling CephFS for our main Ceph cluster (which provides object storage for OpenStack), and one of the obvious questions is what sort of hardware we would need for the MDSs (and how many!). These would be for our users scientific workloads, so they would need to provide reasonably high performance. For reference, we have 3060 6TB OSDs across 51 OSD hosts, and 6 dedicated RGW nodes. The minimum specs are very modest (2-3GB RAM, a tiny amount of disk, similar networking to the OSD nodes), but I'm not sure how much going beyond that is likely to be useful in production. I've also seen it suggested that an SSD-only pool is sensible for the CephFS metadata pool; how big is that likely to get? I'd be grateful for any pointers :) Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

4 3

Re: ceph octopus centos7, containers, cephadm
by Dan van der Ster 23 Oct '20

23 Oct '20

I'm not sure I understood the question. If you're asking if you can run octopus via RPMs on el7 without the cephadm and containers orchestration, then the answer is yes. -- dan On Fri, Oct 23, 2020 at 9:47 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > No clarity on this? > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] ceph octopus centos7, containers, cephadm > > > I am running Nautilus on centos7. Does octopus run similar as nautilus > thus: > > - runs on el7/centos7 > - runs without containers by default > - runs without cephadm by default > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Ceph Octopus and Snapshot Schedules
by Adam Boyhan 23 Oct '20

23 Oct '20

Care to provide anymore detail?

1 0

ceph octopus centos7, containers, cephadm
by Marc Roos 23 Oct '20

23 Oct '20

I am running Nautilus on centos7. Does octopus run similar as nautilus thus: - runs on el7/centos7 - runs without containers by default - runs without cephadm by default

2 2

RFC: Seeking your input on some design documents related to cephadm / Dashboard
by Lenz Grimmer 22 Oct '20

22 Oct '20

Hi all, FYI, we're currently reviewing and discussing several design documents that cover some topics related to "Day 1 installation" and "Day 2 operations" using the functionality provided by cephadm in Ceph Dashboard. We'd like to solicit your input on these plans, so feel free to review and comment on these drafts here (many thanks to Paul Cuzner for kicking this off): docs: Dashboard host management — https://github.com/ceph/ceph/pull/37292 doc/dev/cephadm: Doc defining the design for host maintenance — https://github.com/ceph/ceph/pull/37607 doc/dev/cephadm: high level design for a compliance check feature — https://github.com/ceph/ceph/pull/37519 Thank you for your input! Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

Huge RAM Ussage on OSD recovery
by Ing. Luis Felipe Domínguez Vega 22 Oct '20

22 Oct '20

Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed. Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3 In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that.

4 9

Need help integrating radosgw with keystone for openstack swift
by Bujack, Stefan 22 Oct '20

22 Oct '20

Hello, I am struggling to integrate ceph radosgw as obejctstore in openstack swift via keystone. Could someone please have a look at my configs and help finding the issue? Many thanks ins advance. ceph version 14.2.11 nautilus (stable) [root@ciosmon06 ~]# cat /etc/ceph/ceph.conf [global] fsid = someid mon_initial_members = ciosmon06, ciosmon07, ciosmon08, ciosmon09, ciosmon10 mon_host = 10.254.32.32,10.254.32.33,10.254.32.34,10.254.32.35,10.254.32.36 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.254.32.0/24 cluster_network = 10.254.36.0/24 [client.radosgw.gateway] debug ms = 1 debug rgw = 20 rgw host = 10.254.32.32 rgw keystone verify ssl = false rgw keystone api version = 3 rgw keystone url = https://keystone-intern.desy.de:5000 rgw keystone admin domain = default rgw keystone admin project = service rgw keystone admin user = swift rgw keystone admin password = password rgw s3 auth use keystone = true rgw swift account in url = true rgw keystone implicit tenants = true rgw swift versioning enabled = true rgw enable apis = swift, s3 rgw keystone accepted roles = admin, _member_, member, swiftoperator rgw keystone accepted admin roles = admin, _member_ keyring = /etc/ceph/client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# systemctl status ceph-radosgw(a)rgw.ciosmon06.service ● ceph-radosgw(a)rgw.ciosmon06.service - Ceph rados gateway Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-10-21 17:22:04 CEST; 3min 36s ago Main PID: 3372096 (radosgw) CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw(a)rgw.ciosmon06.service └─3372096 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ciosmon06 --setuser ceph --setgroup ceph Oct 21 17:22:04 ciosmon06.desy.de systemd[1]: Started Ceph rados gateway. [root@ciosmon06 ~]# netstat -ntlup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 3372096/radosgw tcp6 0 0 :::7480 :::* LISTEN 3372096/radosgw [root@ciosmon06 ceph]# ls -la total 32 drwxr-xr-x 2 root root 4096 Oct 21 18:18 . drwxr-xr-x. 93 root root 12288 Sep 24 05:25 .. -rw------- 1 root root 63 Feb 24 2020 ceph.client.admin.keyring -rw-r--r-- 1 root root 1036 Oct 21 18:18 ceph.conf -rw------- 1 root root 71 Oct 21 16:38 client.rgw.ciosmon06.keyring [root@ciosmon06 ~]# cat /etc/ceph/client.rgw.ciosmon06.keyring [client.rgw.ciosmon06] key = somekey [root@ciosmon06 ~]# ceph auth ls client.rgw.ciosmon06 key: somekey caps: [mon] allow rw caps: [osd] allow rwx Openstack ussuri on ubuntu 20.04 root@keystone:~# openstack endpoint list +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+---------------------+---------+-----------+------------------------------------------------------------+ | 0ee9c91af2424e33a91a4c118b693301 | RegionOne | swift | object-store | True | internal | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 1fe9f91db7f942d2a8cb2163bd09772d | RegionOne | keystone | identity | True | admin | https://keystone-intern.desy.de:5000/v3/ | | 4719a266432f45bda380c52486421e62 | RegionOne | swift | object-store | True | public | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | 95a292a5584d4e2c9f8177af3a312494 | RegionOne | keystone | identity | True | public | https://keystone-intern.desy.de:5000/v3/ | | e68b3990e74447bfa35a5d6aa66ca2aa | RegionOne | swift | object-store | True | admin | http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s | | efc1df3cb21344bfb74d8e345e267606 | RegionOne | keystone | identity | True | internal | https://keystone-intern.desy.de:5000/v3/ | root@keystone:~# openstack service list +----------------------------------+-----------+---------------------+ | ID | Name | Type | +----------------------------------+-----------+---------------------+ | 4881557156f844df87d13e427d02b74a | swift | object-store | | 6d928ad49d954a179e37dc17f96c4b1e | keystone | identity | root@keystone:~# openstack user list +----------------------------------+-----------+ | ID | Name | +----------------------------------+-----------+ | ca94085253944ba3803f9d3202172b80 | admin | | ea025c992c224b5f8ca5fea0334eaf15 | swift | +----------------------------------+-----------+ root@keystone:~# openstack role assignment list --names --project service +-------+-------------------+-------+-----------------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+-------------------+-------+-----------------+--------+--------+-----------+ | admin | swift@Default | | service@Default | | | False | +-------+-------------------+-------+-----------------+--------+--------+-----------+ Tests: root@it-build:~# openstack ec2 credentials create +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | access | bc6f1ad9064d4097ad7a323c8368780b | | links | {'self': 'https://keystone-intern.desy.de:5000/v3/users/926c750033e668f0af2330b1c7c72…'} | | project_id | 286f5d2b38ae4595ba9ff8129e754f54 | | secret | c6d9a58220fd4491ba8654d6d3b3c4dc | | trust_id | None | | user_id | 926c750033e668f0af2330b1c7c723a05b86fa393655369fdb1a5622ae65dac8 | +------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ root@it-build:~# /usr/local/bin/aws configure AWS Access Key ID [****************e90c]: bc6f1ad9064d4097ad7a323c8368780b AWS Secret Access Key [****************6ba4]: c6d9a58220fd4491ba8654d6d3b3c4dc Default region name [default]: Default output format [None]: root@it-build:~# /usr/local/bin/aws --endpoint='http://ciosmon06.desy.de:7480/swift/v1/AUTH_$(project_id)s' s3 ls s3:// expected string or bytes-like object [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:24:24.710 7efeabf69700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:24:24.713 7efeabf69700 1 ====== req done req=0x562f3dbb08f0 op status=-2024 http_status=301 latency=0.003s ====== root@keystone:~# openstack role assignment list --names --project 286f5d2b38ae4595ba9ff8129e754f54 | grep swiftoperator | swiftoperator | myusername@closedbeta | | closedbeta@closedbeta | | | False | root@it-build:~# cat myusername-openrc-closedbeta export OS_PROJECT_DOMAIN_NAME=closedbeta export OS_USER_DOMAIN_NAME=closedbeta export OS_PROJECT_NAME=closedbeta export OS_USERNAME=myusername export OS_PASSWORD=password export OS_AUTH_URL=https://keystone-intern.desy.de:5000 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 export OS_VOLUME_API_VERSION=3 root@it-build:~# openstack container list Unrecognized schema in response body. (HTTP 401) (Request-ID: tx000000000000000000005-005f9060f7-26173f-default) [root@ciosmon06 ~]# tail -f /var/log/ceph/ceph-client.rgw.ciosmon06.log 2020-10-21 18:25:26.868 7efea9f65700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:26.868 7efea9f65700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== starting new request req=0x562f3dbb08f0 ===== 2020-10-21 18:25:27.088 7efeadf6d700 1 ====== req done req=0x562f3dbb08f0 op status=0 http_status=401 latency=0s ====== Greets Stefan Bujack

2 2

6 PG's stuck not-active, remapped
by Mac Wynkoop 22 Oct '20

22 Oct '20

We recently did some work on the Ceph cluster, and a few disks ended up offline at the same time. There are now 6 PG's that are stuck in a "remapped" state, and this is all of their recovery states: *recovery_state: 0: name: Started/Primary/WaitActingChangeenter_time: 2020-10-21 18:48:02.034430comment: waiting for pg acting set to change1: name: Startedenter_time: 2020-10-21 18:48:01.752957* Any ideas? Mac Wynkoop

2 2

Re: Huge RAM Ussage on OSD recovery
by Ing. Luis Felipe Domínguez Vega 21 Oct '20

21 Oct '20

El 2020-10-21 10:08, Mark Nelson escribió: > On 10/21/20 7:54 AM, Ing. Luis Felipe Domínguez Vega wrote: >> El 2020-10-21 08:43, Mark Nelson escribió: >>> Theoretically we shouldn't be spiking memory as much these days >>> during >>> recovery, but the code is complicated and it's tough to reproduce >>> these kinds of issues in-house. If you happen to catch it in the >>> act, >>> do you see the pglog mempool stats also spiking up? >>> >>> >>> Mark >>> >>> >>> On 10/21/20 2:34 AM, Dan van der Ster wrote: >>>> Hi, >>>> >>>> This might be the pglog issue which has been coming up a few times >>>> on the list. >>>> If the OSD cannot boot without going OOM, you might have success by >>>> trimming the pglog, e.g. search this list for "ceph-objectstore-tool >>>> --op trim-pg-log" for some recipes. The thread "OSDs taking too much >>>> memory, for pglog" in particular might help. >>>> >>>> Cheers, Dan >>>> >>>> >>>> >>>> On Tue, Oct 20, 2020 at 11:57 PM Ing. Luis Felipe Domínguez Vega >>>> <luis.dominguez(a)desoft.cu> wrote: >>>>> Hi, today mi Infra provider has a blackout, then the Ceph was try >>>>> to >>>>> recover but are in an inconsistent state because many OSD can >>>>> recover >>>>> itself because the kernel kill it by OOM. Even now one OSD that was >>>>> OK, >>>>> go down by OOM killed. >>>>> >>>>> Even in a server with 32GB RAM the OSD use ALL that and never >>>>> recover, i >>>>> think that can be a memory leak, ceph version octopus 15.2.3 >>>>> >>>>> In: https://pastebin.pl/view/59089adc >>>>> You can see that buffer_anon get 32GB, but why?? all my cluster is >>>>> down >>>>> because that. >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> this https://pastebin.pl/view/59089adc is almost the OSD going to be >> killed by OOM >> > > Ok, that is very interesting! The OSD memory autotuning code shrank > the caches to be almost nothing to try and compensate for the huge > growth in buffer_anon (and to a lesser extent osd_pglog) usage but > obviously couldn't do anything with that much memory being used. Any > chance you could create a tracker ticket and paste the memory pool > info in along with ceph version/etc? > > > https://tracker.ceph.com/ > > > Mark Thanks, https://tracker.ceph.com/issues/47929

1 0

Difference between node exporter and ceph exporter data
by Seena Fallah 21 Oct '20

21 Oct '20

Hi all, There is a huge difference between node exporter and ceph exporter (prometheus mgr module) data. For example I see there is a 120MB/s write on my disk from node exporter but ceph exporter says it is 22MB! Also for latency and IOPS and... Which one is reliable? Thanks.

1 0

Re: pool pgp_num not updated
by Mac Wynkoop 21 Oct '20

21 Oct '20

Right, both Norman and I set the pg_num before the pgp_num. For example, here is my current pool settings: *"pool 40 '*redacted*.rgw.buckets.data' erasure size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 2048 pgp_num 1024 pgp_num_target 2048 last_change 8458830 lfor 0/0/8445757 flags hashpspool,ec_overwrites,nodelete,backfillfull stripe_width 24576 fast_read 1 application rgw"* So, when I set: "*ceph osd pool set hou-ec-1.rgw.buckets.data pgp_num 2048*" it returns: "*set pool 40 pgp_num to 2048*" But upon checking the pool details again: "*pool 40 '*redacted*.rgw.buckets.data' erasure size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 2048 pgp_num 1024 pgp_num_target 2048 last_change 8458870 lfor 0/0/8445757 flags hashpspool,ec_overwrites,nodelete,backfillfull stripe_width 24576 fast_read 1 application rgw*" and the pgp_num value does not increase. Am I just doing something totally wrong? Thanks, Mac Wynkoop On Tue, Oct 6, 2020 at 2:32 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > pg_num and pgp_num need to be the same, not? > > 3.5.1. Set the Number of PGs > > To set the number of placement groups in a pool, you must specify the > number of placement groups at the time you create the pool. See Create a > Pool for details. Once you set placement groups for a pool, you can > increase the number of placement groups (but you cannot decrease the > number of placement groups). To increase the number of placement groups, > execute the following: > > ceph osd pool set {pool-name} pg_num {pg_num} > > Once you increase the number of placement groups, you must also increase > the number of placement groups for placement (pgp_num) before your > cluster will rebalance. The pgp_num should be equal to the pg_num. To > increase the number of placement groups for placement, execute the > following: > > ceph osd pool set {pool-name} pgp_num {pgp_num} > > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/s… > > -----Original Message----- > To: norman > Cc: ceph-users > Subject: [ceph-users] Re: pool pgp_num not updated > > Hi everyone, > > I'm seeing a similar issue here. Any ideas on this? > Mac Wynkoop, > > > > On Sun, Sep 6, 2020 at 11:09 PM norman <norman.kern(a)gmx.com> wrote: > > > Hi guys, > > > > When I update the pg_num of a pool, I found it not worked(no > > rebalanced), anyone know the reason? Pool's info: > > > > pool 21 'openstack-volumes-rs' replicated size 3 min_size 2 crush_rule > > 21 object_hash rjenkins pg_num 1024 pgp_num 512 pgp_num_target 1024 > > autoscale_mode warn last_change 85103 lfor 82044/82044/82044 flags > > hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd > > removed_snaps > > [1~1e6,1e8~300,4e9~18,502~3f,542~11,554~1a,56f~1d7] > > pool 22 'openstack-vms-rs' replicated size 3 min_size 2 crush_rule 22 > > object_hash rjenkins pg_num 512 pgp_num 512 pg_num_target 256 > > pgp_num_target 256 autoscale_mode warn last_change 84769 lfor > > 0/0/55294 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 > > application rbd > > > > The pgp_num_target is set, but pgp_num not set. > > > > I have scale out new OSDs and is backfilling before setting the value, > > > is it the reason? > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > >

4 11

Fwd: [lca-announce] linux.conf.au 2021 - Call for Sessions and Miniconfs Open
by Tim Serong 21 Oct '20

21 Oct '20

The best F/OSS conference in the southern hemisphere is back again, virtualized, January 23-25. The CFP is open until November 6. Submit early, submit often! ;-) -------- Forwarded Message -------- Subject: [lca-announce] linux.conf.au 2021 - Call for Sessions and Miniconfs Open Date: Thu, 15 Oct 2020 20:43:24 +1000 From: linux.conf.au Announcements <lca-announce(a)lists.linux.org.au> Reply-To: lca-announce(a)lists.linux.org.au To: lca-announce(a)lists.linux.org.au We're excited! The linux.conf.au 2021 Call for Sessions and Call for Miniconfs are now open. They will stay open until 6th November 2020 Anywhere on Earth (AoE) (https://en.wikipedia.org/wiki/Anywhere_on_Earth) This is only 3 weeks away - so don't delay, get your talks in early! Our theme is "So what's next?". We all know we're living through unprecedented change and uncertain times. How can open source play a role in creating, helping and adapting to this ongoing change? What new developments in software and coding can we look forward to in 2021 and beyond? If you have ideas or developments you'd like to share with the open source community at linux.conf.au, we'd love to hear from you. Call for Sessions The main conference runs on Sunday 24 and Monday 25 January, with multiple streams catering for a wide range of interest areas. We invite you to submit a session (https://linux.conf.au/programme/sessions/) proposal for a talk. Talks are generally 35-45 minute presentations on a single topic presented in lecture format. Call for Miniconfs Miniconfs are dedicated day-long streams focusing on single topics, creating a more immersive experience for delegates than a session. We encourage you to get creative with how you could deliver your Miniconf virtually! Running a Miniconf (https://linux.conf.au/programme/miniconfs/) is a great way to gain experience, provide exposure for your project or topic, and raise your professional profile. They're a crowd favourite and an awesome way to kick off the conference. Miniconfs will run on Saturday 23 January, before the main conference commences on Sunday. No need to book flights or hotels Don't forget: the conference will be a fully online, virtual experience. This means our speakers will be beaming in from their own homes or workplaces. The organising team will be able to help speakers with their tech set-up. Each accepted presenter will have a tech check prior to the event to smooth out any difficulties and there will be an option to pre-record presentations. Have we piqued your interest? You can find out how to submit your session or miniconf proposals at https://linux.conf.au/programme/proposals/. If you have any other questions you can contact us via email at contact(a)lca2021.linux.org.au. Timeline 16th October: Call for sessions opens 6th November: Call for sessions closes January 23 2021: Miniconfs! January 24-25 2021: Main conference presentations Tickets will go on sale in the coming weeks. We'll keep you posted. We're looking forward to reading your submissions. linux.conf.au 2021 Organising Team About linux.conf.au 2021 Running since 1999, linux.conf.au is the largest linux and open source conference in the Asia-Pacific region. The conference provides deeply technical presentations from industry leaders and experts on a wide array of subjects relating to open source projects, data and open government and community engagement. --- Read this online at https://lca2021.linux.org.au/news/call-for-sessions-miniconfs-open/ _______________________________________________ lca-announce mailing list lca-announce(a)lists.linux.org.au http://lists.linux.org.au/mailman/listinfo/lca-announce

1 0

How to see dprintk output
by 展荣臻（信泰） 21 Oct '20

21 Oct '20

Hi, There are many dprintk call in crush/mapper.c and crush/builder.c,I want to debug crush algorithm. How I to see output of dprintk?

1 0

Re: Huge RAM Ussage on OSD recovery
by Ing. Luis Felipe Domínguez Vega 21 Oct '20

21 Oct '20

El 2020-10-20 23:17, Anthony D'Atri escribió: >> On Oct 20, 2020, at 6:23 PM, Ing. Luis Felipe Domínguez Vega >> <luis.dominguez(a)desoft.cu> wrote: >> >> El 2020-10-20 19:33, Anthony D'Atri escribió: >>> You have a *lot* of peering and recovery going on. >>> Write a script that monitors available memory on the system and >>> restarts the OSD process using the most when it crosses some >>> threshold. Run that on all OSD nodes. OSDs will come up, make some >>> progress, get restarted, but eventually they’ll sync up. >>>> On Oct 20, 2020, at 2:57 PM, Ing. Luis Felipe Domínguez Vega >>>> <luis.dominguez(a)desoft.cu> wrote: >>>> Hi, today mi Infra provider has a blackout, then the Ceph was try to >>>> recover but are in an inconsistent state because many OSD can >>>> recover itself because the kernel kill it by OOM. Even now one OSD >>>> that was OK, go down by OOM killed. >>>> Even in a server with 32GB RAM the OSD use ALL that and never >>>> recover, i think that can be a memory leak, ceph version octopus >>>> 15.2.3 >>>> In: https://pastebin.pl/view/59089adc >>>> You can see that buffer_anon get 32GB, but why?? all my cluster is >>>> down because that. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> That's the only solution??? > > I didn’t say that. If you don’t like it, don’t do it. You can also > try eras’ idea. You’ve offered almost no detail about your cluster or > hardware. > > >> there is nothing to limit the OSD resource usage on recover? Sorry, what is the era's idea? And i can send my cluster info, what do you need to try a solution (already i'm running your suggestion of a script that restart the OSD on high memory)

1 0

Ceph OIDC Integration
by technical＠zylacomputing.com 21 Oct '20

21 Oct '20

Hello, we have integrated Ceph's RGW with LDAP and have authenticated users using the mail attribute successfully. We would like to shift to SSO and are evaluating the new OIDC feature in Ceph together with dexIdP with an LDAP connector as an upstream IdP. We are trying to understand the flow of the user authentication and how it will effect my current LDAP users buckets which are already created in Ceph as LDAP users. Will the Ceph RGW be able to pass the token to be verified to the IdP and what type of user will then be created in Ceph? Is this the intended way of OIDC integration? Thanks for any assistance

2 7

Problems with ceph command - Octupus - Ubuntu 16.04
by Emanuel Alejandro Castelli 20 Oct '20

20 Oct '20

Hello I'm facing an issue with ceph. I cannot run any ceph command. It literally hangs. I need to hit CTRL-C to get this: ^CCluster connection interrupted or timed out This is on Ubuntu 16.04. Also, I use Graphana with Prometheus to get information from the cluster, but now there is no data to graph. Any clue? BQ_BEGIN cephadm version BQ_END BQ_BEGIN INFO:cephadm:Using recent ceph image ceph/ceph:v15 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) BQ_END cephadm ls [ { "style": "cephadm:v1", "name": "mon.osswrkprbe001", "fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d", "systemd_unit": "ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mon.osswrkprbe001", "enabled": true, "state": "running", "container_id": "afbe6ef76198bf05ec972e832077849d4a4438bd56f2e177aeb9b11146577baf", "container_image_name": "docker.io/ceph/ceph:v15.2.1", "container_image_id": "bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2", "version": "15.2.1", "started": "2020-10-19T19:03:16.759730", "created": "2020-09-04T23:30:30.250336", "deployed": "2020-09-04T23:48:20.956277", "configured": "2020-09-04T23:48:22.100283" }, { "style": "cephadm:v1", "name": "mgr.osswrkprbe001", "fsid": "56820176-ae5b-4e58-84a2-442b2fc03e6d", "systemd_unit": "ceph-56820176-ae5b-4e58-84a2-442b2fc03e6d(a)mgr.osswrkprbe001", "enabled": true, "state": "running", "container_id": "1737b2cf46310025c0ae853c3b48400320fb35b0443f6ab3ef3d6cbb10f460d8", "container_image_name": "docker.io/ceph/ceph:v15.2.1", "container_image_id": "bc83a388465f0568dab4501fb7684398dca8b50ca12a342a57f21815721723c2", "version": "15.2.1", "started": "2020-10-19T20:43:38.329529", "created": "2020-09-04T23:30:31.110341", "deployed": "2020-09-04T23:47:41.604057", "configured": "2020-09-05T00:00:21.064246" } ] Thank you in advance. Saludos, EMANUEL CASTELLI Arquitecto de Información - Gerencia OSS C: (+549) 116707-4107 | Interno: 1325 | T-Phone: 7510-1325 | ecastelli(a)telecentro.net.ar Lavardén 157 1er piso. CABA (C1437FBC)

2 6

Recommended settings for PostgreSQL
by Gencer W. Genç 20 Oct '20

20 Oct '20

Hi, I have an existing few RBDs. I would like to create a new RBD Image for PostgreSQL. Do you have any suggestions for such use cases? For example; Currently defaults are: Object size (4MB) and Stripe Unit (None) Features: Deep flatten + Layering + Exclusive Lock + Object Map + FastDiff Should I use as is or should I use 16KB of object size and different sets of features for PostgreSQL? Thanks, Gencer.

5 7

Mon DB compaction MON_DISK_BIG
by Szabo, Istvan (Agoda) 20 Oct '20

20 Oct '20

Hi, I've received a warning today morning: HEALTH_WARN mons monserver-2c01,monserver-2c02,monserver-2c03 are using a lot of disk space MON_DISK_BIG mons monserver-2c01,monserver-2c02,monserver-2c03 are using a lot of disk space mon.monserver-2c01 is 15.3GiB >= mon_data_size_warn (15GiB) mon.monserver-2c02 is 15.3GiB >= mon_data_size_warn (15GiB) mon.monserver-2c03 is 15.3GiB >= mon_data_size_warn (15GiB) It hits the 15GB so I've restarted all the 3 mons, it triggered compaction. I've also ran this command: ceph tell mon.`hostname -s` compact on the first node, but it wents down only to 13GB. du -sch /var/lib/ceph/mon/ceph-monserver-2c01/store.db/ 13G /var/lib/ceph/mon/ceph-monserver-2c01/store.db/ 13G total Anything else I can do to reduce it? Luminous 12.2.8 is the version. Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 4

Re: Recommended settings for PostgreSQL
by Gencer W. Genç 20 Oct '20

20 Oct '20

Marc Roos wrote: > > In the past I see some good results (benchmark & > > latencies) for MySQL and PostgreSQL. However, I've always used > > 4MB object size. Maybe i can get much better > > performance on smaller object size. Haven't tried actually. > > Did you tune mysql / postgres for this setup? Did you have a default > ceph rbd setup? Yes, I had to tune some settings on PostgreSQL. Especially on: synchronous_commit = off I have a default RBD settings. Do you have any recommendation? Thanks, Gencer.

1 0

OSD host count affecting available pool size?
by Dallas Jones 19 Oct '20

19 Oct '20

Hi, Ceph brain trust: I'm still trying to wrap my head around some capacity planning for Ceph, and I can't find a definitive answer to this question in the docs (at least one that penetrates my mental haze)... Does the OSD host count affect the total available pool size? My cluster consists of three 12-bay Dell PowerEdge machines running reflashed PERCs to make each SAS drive individually addressable. Each node is running 10 OSDs. Is Ceph limiting the max available pool size because all of my OSDs are being hosted on just three nodes? If I had 30 OSDs running across ten nodes instead, a node failure would result in just three OSDs dropping out instead of ten. Is there any rationale to this thinking, or am I trying to manufacture a solution to a problem I still don't understand? Thanks, Dallas

2 2

Re: Recommended settings for PostgreSQL
by Gencer W. Genç 19 Oct '20

19 Oct '20

Yes, I had to tune some settings on PostgreSQL. Especially on: synchronous_commit = off I have a default RBD settings. Do you have any recommendation? Thanks, Gencer. On 19.10.2020 12:49:51, Marc Roos <m.roos(a)f1-outsourcing.eu> wrote: > In the past I see some good results (benchmark & latencies) for MySQL and PostgreSQL. However, I've always used > 4MB object size. Maybe i can get much better performance on smaller object size. Haven't tried actually. Did you tune mysql / postgres for this setup? Did you have a default ceph rbd setup?

1 0

RGW with HAProxy
by Seena Fallah 19 Oct '20

19 Oct '20

Hi When I use haproxy with keep-alive mode to rgws, haproxy gives many responses like this! Is there any problem with keep-alive mode in rgw? Using nautilus 14.2.9 with beast frontend.

1 0

Module 'cephadm' has failed: cephadm exited with an error code: 2, stderr:usage: rm-daemon [-h] --name NAME --fsid FSID [--force] [--force-delete-data]
by 周凡夫 19 Oct '20

19 Oct '20

Hello, I had ERRs in using Cephadm for my company. This an important question for me and my company. Hope you can post it. Thanks a lot. Fanfu

1 0

Re: multiple OSD crash, unfound objects
by Chad William Seys 16 Oct '20

16 Oct '20

This problem may also be related to the below unsolved issue, which specifically mentions 'unfound' objects. Sadly, there is probably nothing in the report which will help with your troubleshooting. https://tracker.ceph.com/issues/44286 C.

1 0

radosgw bucket subdomain with tls
by Marc Roos 15 Oct '20

15 Oct '20

I enabled a certificate on my radosgw, but I think I am running into the problem that the s3 clients are accessing the buckets like bucket.rgw.domain.com. Which fails my cert rgw.domain.com. Is there any way to configure that only rgw.domain.com is being used?

3 3

Cannot ingest data after quota full and modify quota
by St-Germain, Sylvain (SSC/SPC) 15 Oct '20

15 Oct '20

Good morning all, I don't know if this happened to someone, but recently a user ran out of their quota for the number of objects in a bucket. The only sign I could see in the logs (tail -f /var/log/ceph/ceph-rgw-dao-wkr-01.rgw0.log) was the following: 2020-10-01T03:07:02.098+0000 7fd872bf6700 1 ====== req done req=0x7fd8cb13a8a0 op status=-2026 http_status=413 latency=17.304220706s ====== When I removed the User limits the error went from 413 to 422 in the logs. radosgw-admin quota set --quota-scope=bucket --uid=smithj --max-objects=-1 2020-10-15T11:58:41.661+0000 7f0c79b51700 1 ====== req done req=0x7f0cb7f5f8a0 op status=-2018 http_status=422 latency=0.001000013s ====== I have tried a lot of radosgw-admin command, but so far the error persists. Do you have any ideas ? Thx ! Sylvain

1 0

Using crushtool reclassify to insert device class into existing crush map
by Mathias Lindberg 15 Oct '20

15 Oct '20

Hi, We are trying to introduce SSD/NvME OSD’s and to prevent data moving from current (hdd based) OSD’s while also having erasure coded pools we could not just simply change the erasure coding profile or create a new one and just apply it to the pool. Reading this list and other posts on forums it was suggested to use the crushtool reclassify to insert device classes into the current crush rules and load that crush map manually. Not having edited the crush map in this or any other fashion earlier I would appreciate very much if I someone could verify if I have done the reclassify correctly. Thank you, [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --reclassify --reclassify-root default hdd -o crush_comp_corr.c classify_root default (-1) as hdd renumbering bucket -1 -> -29 renumbering bucket -27 -> -30 renumbering bucket -25 -> -31 renumbering bucket -23 -> -32 renumbering bucket -21 -> -33 renumbering bucket -19 -> -34 renumbering bucket -17 -> -35 renumbering bucket -15 -> -36 renumbering bucket -13 -> -37 renumbering bucket -11 -> -38 renumbering bucket -9 -> -39 renumbering bucket -7 -> -40 renumbering bucket -5 -> -41 renumbering bucket -3 -> -42 [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare crush_comp_corr.c rule 0 had 0/10240 mismatched mappings (0) rule 6 had 0/10240 mismatched mappings (0) rule 7 had 0/4096 mismatched mappings (0) rule 8 had 0/4096 mismatched mappings (0) rule 9 had 0/4096 mismatched mappings (0) rule 10 had 0/4096 mismatched mappings (0) rule 11 had 0/4096 mismatched mappings (0) rule 12 had 0/4096 mismatched mappings (0) rule 13 had 0/4096 mismatched mappings (0) maps appear equivalent Regards, Mathias Lindberg Tel: +46 (0)31 7723059 Mob: +46 (0)723 526107 Mathias Lindberg mathlin(a)chalmers.se

2 1