December 2019 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

atime with cephfs
by Oliver Freyermuth 04 Dec '20

04 Dec '20

Dear Cephers, we are currently mounting CephFS with relatime, using the FUSE client (version 13.2.6): ceph-fuse on /cephfs type fuse.ceph-fuse (rw,relatime,user_id=0,group_id=0,allow_other) For the first time, I wanted to use atime to identify old unused data. My expectation with "relatime" was that the access time stamp would be updated less often, for example, only if the last file access was >24 hours ago. However, that does not seem to be the case: ---------------------------------------------- $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... $ cat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root > /dev/null $ sync $ stat /cephfs/grid/atlas/atlaslocalgroupdisk/rucio/group/phys-higgs/ed/cb/group.phys-higgs.17620861._000004.HSM_common.root ... Access: 2019-04-10 15:50:04.975959159 +0200 Modify: 2019-04-10 15:50:05.651613843 +0200 Change: 2019-04-10 15:50:06.141006962 +0200 ... ---------------------------------------------- I also tried this via an nfs-ganesha mount, and via a ceph-fuse mount with admin caps, but atime never changes. Is atime really never updated with CephFS, or is this configurable? Something as coarse as "update at maximum once per day only" would be perfectly fine for the use case. Cheers, Oliver

4 6

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

RBD logs
by 陈旭 04 Jun '20

04 Jun '20

Hi guys, I deploy an efk cluster and use ceph as block storage in kubernetes, but RBD write iops sometimes becomes zero and last for a few minutes. I want to check logs about RBD so I add some config to ceph.conf and restart ceph. Here is my ceph.conf: [global] fsid = 53f4e1d5-32ce-4e9c-bf36-f6b54b009962 mon_initial_members = db-16-4-hzxs, db-16-5-hzxs, db-16-6-hzxs mon_host = 10.25.16.4,10.25.16.5,10.25.16.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd pool default size = 3 [client] debug rbd = 20 debug rbd mirror = 20 debug rbd replay = 20 log file = /var/log/ceph/client_rbd.log I can not get any logs in /var/log/ceph/client_rbd.log. I also try to execute 'ceph daemon osd.* config set debug_rbd 20’ and there is also no related logs in ceph-osd.log. How can I get useful logs about this question or How can I analyze this problem? Look forward to your reply. Thanks ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 声明：此邮件可能包含依图公司保密或特权信息，并且仅应发送至有权接收该邮件的收件人。如果您无权收取该邮件，您应当立即删除该邮件并通知发件人，您并被禁止传播、分发或复制此邮件以及附件。对于此邮件可能携带的病毒引起的任何损害，本公司不承担任何责任。此外，本公司不保证已正确和完整地传输此信息，也不接受任何延迟收件的赔偿责任。 ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notice: This email may contain confidential or privileged information of Yitu and was sent solely to the intended recipients. If you are unauthorized to receive this email, you should delete the email and contact the sender immediately. Any unauthorized disclosing, distribution, or copying of this email and attachment thereto is prohibited. Yitu does not accept any liability for any loss caused by possibly viruses in this email. E-mail transmission cannot be guaranteed to be secure or error-free and Yitu is not responsible for any delayed transmission.

2 1

Help! ceph-mon is blocked after shutting down and ip address changed
by occj＠qq.com 04 Jun '20

04 Jun '20

ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) os :CentOS Linux release 7.7.1908 (Core) single node ceph cluster with 1 mon,1mgr,1 mds,1rgw and 12osds , but only cephfs is used. ceph -s is blocked after shutting down the machine (192.168.0.104), then ip address changed to 192.168.1.6 I created the monmap with monmap tool and update the ceph.conf , hosts file and then start ceph-mon. and the ceph-mon log: ... 2019-12-11 08:57:45.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1285.14s 2019-12-11 08:57:50.170 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1290.14s 2019-12-11 08:57:55.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1295.14s 2019-12-11 08:58:00.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1300.14s 2019-12-11 08:58:05.172 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1305.14s 2019-12-11 08:58:10.171 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1310.14s 2019-12-11 08:58:15.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1315.14s 2019-12-11 08:58:20.173 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1320.14s 2019-12-11 08:58:25.174 7f952cdac700 1 mon.ceph-node1 at 0(leader).mds e34 no beacon from mds.0.10 (gid: 4384 addr: [v2:192.168.0.104:6898/4084823750,v1:192.168.0.104:6899/4084823750] state: up:active) since 1325.14s ... I changed IP back to 192.168.0.104 yeasterday, but all the same. # cat /etc/ceph/ceph.conf [client.libvirt] admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor log file = /var/log/ceph/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor [client.rgw.ceph-node1.rgw0] host = ceph-node1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-node1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-node1.rgw0.log rgw frontends = beast endpoint=192.168.1.6:8080 rgw thread pool size = 512 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.1.0/24 fsid = e384e8e6-94d5-4812-bfbb-d1b0468bdef5 mon host = [v2:192.168.1.6:3300,v1:192.168.1.6:6789] mon initial members = ceph-node1 osd crush chooseleaf type = 0 osd pool default crush rule = -1 public network = 192.168.1.0/24 [osd] osd memory target = 7870655146

2 1

ceph with rdma can not mount with kernel
by 李亚锋 29 May '20

29 May '20

2 1

Ceph on CentOS 8?
by Jan Kasprzak 29 May '20

29 May '20

Hello, Ceph users, does anybody use Ceph on recently released CentOS 8? Apparently there are no el8 packages neither at download.ceph.com, nor in the native CentOS package tree. I am thinking about upgrading my cluster to C8 (because of other software running on it apart from Ceph). Do el7 packages simply work? Can they be rebuilt using rpmbuild --rebuild? Or is running Ceph on C8 more complicated than that? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | sir_clive> I hope you don't mind if I steal some of your ideas? laryross> As far as stealing... we call it sharing here. --from rcgroups

9 8

rbd image naming convention
by Palanisamy 29 May '20

29 May '20

Hello Team, We've integrated Ceph cluster storage with Kubernetes and provisioning volumes through rbd-provisioner. When we're creating volumes from yaml files in Kubernetes, pv > pvc > mounting to pod, In kubernetes end pvc are showing as meaningful naming convention as per yaml file defined. But in ceph cluster, rbd image name is creating with dynamic uid. During troubleshooting time, this will be tedious to find exact rbd image. Please find the provisioning logs in below pasted snippet. kubectl get pods,pv,pvc NAME READY STATUS RESTARTS AGE pod/sleepypod 1/1 Running 0 4m9s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO Delete Bound default/test-dyn-pvc ceph-rbd 4m9s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-dyn-pvc Bound pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 1Gi RWO ceph-rbd 4m11s *rbd-provisioner logs* I1121 10:59:15.009012 1 provision.go:132] successfully created rbd image "kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2" I1121 10:59:15.009092 1 controller.go:1087] provision "default/test-dyn-pvc" class "ceph-rbd": volume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" provisioned I1121 10:59:15.009138 1 controller.go:1101] provision "default/test-dyn-pvc" class "ceph-rbd": trying to save persistentvvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" I1121 10:59:15.020418 1 controller.go:1108] provision "default/test-dyn-pvc" class "ceph-rbd": persistentvolume "pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5" saved I1121 10:59:15.020476 1 controller.go:1149] provision "default/test-dyn-pvc" class "ceph-rbd": succeeded I1121 10:59:15.020802 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"test-dyn-pvc", UID:"cd37d2d6-cecc-4a05-9736-c8d80abde7f5", APIVersion:"v1", ResourceVersion:"24545639", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-cd37d2d6-cecc-4a05-9736-c8d80abde7f5 *rbd image details in Ceph cluster end* rbd -p kube ls --long NAME SIZE PARENT FMT PROT LOCK kubernetes-dynamic-pvc-f4eac482-0c4d-11ea-8d70-8a582e0eb4e2 1 GiB 2 is there way to setup proper naming convention for rbd image as well during kubernetes deployment itself. Kubernetes version: v1.15.5 Ceph cluster version: 14.2.2 nautilus (stable) *Best Regards,* *Palanisamy*

2 2

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

RGWReshardLock::lock failed to acquire lock ret=-16
by Josh Haft 12 Mar '20

12 Mar '20

Hi, Currently running Mimic 13.2.5. We had reports this morning of timeouts and failures with PUT and GET requests to our Ceph RGW cluster. I found these messages in the RGW log: RGWReshardLock::lock failed to acquire lock on bucket_name:bucket_instance ret=-16 NOTICE: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is still resharding, please retry Which were preceded by many of these, which I think are normal/expected. check_bucket_shards: resharding needed: stats.num_objects=6415879 shard max_objects=6400000 Our RGW cluster sits behind haproxy which notified me approx 90 seconds after the first 'resharding needed' message that no backends were available. It appears this dynamic reshard process caused the RGWs to lock up for a period of time. Roughly 2 minutes later the reshard error messages stop and operation returns to normal. Looking back through previous RGW logs, I see a similar event from about a week ago, on the same bucket. We have several buckets with shard counts exceeding 1k (this one only has 128), and much larger object counts, so clearly this isn't the first time dynamic sharding has been invoked on this cluster. Has anyone seen this? I expect it will come up again, and can turn up debugging if that'll help. Thanks for any assistance! Josh

1 1

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

ceph-mon using 100% CPU after upgrade to 14.2.5
by Bryan Stillwell 24 Feb '20

24 Feb '20

On our test cluster after upgrading to 14.2.5 I'm having problems with the mons pegging a CPU core while moving data around. I'm currently converting the OSDs from FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the OSDs, and then recreating them with ceph-volume lvm batch. This seems too get the ceph-mon process into a state where it pegs a CPU core on one of the mons: 1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72 ceph-mon Has anyone else run into this with 14.2.5 yet? I didn't see this problem while the cluster was running 14.2.4. Thanks, Bryan

5 8

slow using ISCSI - Help-me
by Gesiel Galvão Bernardes 19 Feb '20

19 Feb '20

Hi, I am having an unusual slowdown using VMware with ISCSI gws. I have two ISCSI gateways with two RBD images. I have checked the following in the logs: Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:26.040 969 [INFO] alua_implicit_transition:557 rbd/pool1.vmware_iscsi1: Lock acquisition operation is already in process.2019-12-24 09:00:26.973 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:26 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.099 969 [WARN] tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_notify_lock_lost:201 rbd/pool1.vmware_iscsi1: Async lock drop. Old state 1 Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: 2019-12-24 09:00:28.824 969 [INFO] alua_implicit_transition:562 rbd/pool1.vmware_iscsi1: Starting lock acquisition operation.2019-12-24 09:00:28.990 969 [WARN] tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Dec 24 09:00:28 ceph-iscsi2 tcmu-runner: tcmu_rbd_lock:744 rbd/pool1.vmware_iscsi1: Acquired exclusive lock. Can anyone help-me please? Gesiel

2 14

ceph status reports: slow ops - this is related to long running process /usr/bin/ceph-osd
by Thomas 18 Feb '20

18 Feb '20

Hi, ceph status reports: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available 4 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 59 pgs inactive, 16 pgs peering Degraded data redundancy: 597/153910758 objects degraded (0.000%), 2 pgs degraded, 1 pg undersized Degraded data redundancy (low space): 23 pgs backfill_toofull 1 pgs not deep-scrubbed in time 4 pgs not scrubbed in time 3 pools have too many placement groups 164 slow requests are blocked > 32 sec 1082 stuck requests are blocked > 4096 sec 1490 slow ops, oldest one blocked for 19711 sec, daemons [osd,0,osd,175,osd,186,osd,5,osd,6,osd,63,osd,68,osd,9,mon,ld5505,mon,ld5506]... have slow ops. services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 5h) mgr: ld5507(active, since 5h), standbys: ld5506, ld5505 mds: pve_cephfs:0/1, 1 failed osd: 419 osds: 416 up, 416 in; 6024 remapped pgs data: pools: 6 pools, 8864 pgs objects: 51.30M objects, 196 TiB usage: 594 TiB used, 907 TiB / 1.5 PiB avail pgs: 0.666% pgs not active 597/153910758 objects degraded (0.000%) 52964415/153910758 objects misplaced (34.412%) 5954 active+remapped+backfill_wait 2786 active+clean 40 active+remapped+backfilling 35 activating 23 active+remapped+backfill_wait+backfill_toofull 16 peering 7 activating+remapped 1 activating+undersized+degraded 1 active+clean+scrubbing 1 active+recovering+degraded io: client: 3.5 KiB/s wr, 0 op/s rd, 0 op/s wr recovery: 551 MiB/s, 137 objects/s I'm concerned about the slow ops on osd.0 and osd.9. On the relevant OSD node I can see 2 relevant services running for hours: ceph 14795 1 99 09:58 ? 08:49:22 /usr/bin/ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph ceph 15394 1 99 09:58 ? 07:10:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph In the relevant osd log I can find similar messages: root@ld5505:~# tail -f /var/log/ceph/ceph-osd.0.log 2019-10-08 15:35:32.830 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:33.806 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:34.842 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:35.862 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) root@ld5505:~# tail -f /var/log/ceph/ceph-osd.9.log 2019-10-08 15:35:38.822 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:39.854 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:40.850 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:41.862 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) Question: How can I analyse and solve the issue with slow ops? THX

3 2

RBD-mirror instabilities
by Oliver Freyermuth 15 Feb '20

15 Feb '20

Dear Cephalopodians, running 13.2.6 on the source cluster and 14.2.5 on the rbd mirror nodes and the target cluster, I observe regular failures of rbd-mirror processes. With failures, I mean that traffic stops, but the daemons are still listed as active rbd-mirror daemons in "ceph -s", and the daemons are still running. This comes in sync with a hefty load of below messages in the mirror logs. This happens "sometimes" when some OSDs go down and up in the target cluster (which happens each night since the disks in that cluster shortly go offline during "online" smart self-tests - that's a problem in itself, but it's a cluster built from hardware that would have been trashed otherwise). The rbd daemons keep running in any case, but synchronization stops. If not all rbd mirror daemons have failed (we have three running, and it usually does not hit all of them), the "surviving" seem(s) not to take care of the images the other daemons had locked. Right now, I am eyeing with a "quick solution" of regularly restarting the rbd-mirror daemons, but if there are any good ideas on which debug info I could collect to get this analyzed and fixed, that would of course be appreciated :-). Cheers, Oliver ----------------------------------------------- 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_process_entry_safe: failed to commit journal event: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:51.379 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb968d00 [2/aabba863-89fd-4ea5-bb8c-0f417225d394] handle_replay_complete: replay encountered an error: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_process_entry_safe: failed to commit journal event: (125) Operation canceled 2019-12-24 02:08:54.392 7f31c530e700 -1 rbd::mirror::ImageReplayer: 0x559dcb87bb00 [2/23699357-a611-4557-9d73-6ff5279da991] handle_replay_complete: replay encountered an error: (125) Operation canceled 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dce2e05b0 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:08:55.707 7f31ea358700 -1 rbd::mirror::image_replayer::GetMirrorImageIdRequest: 0x559dcf47ee70 handle_get_image_id: failed to retrieve image id: (108) Cannot send after transport endpoint shutdown ... 2019-12-24 02:08:55.716 7f31f5b6f700 -1 rbd::mirror::ImageReplayer: 0x559dcb997680 [2/f8218221-6608-4a2b-8831-84ca0c2cb418] operator(): start failed: (108) Cannot send after transport endpoint shutdown 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0577bd16-acc4-4e9a-81f0-c698a24f8771: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=05bd4cca-a561-4a5c-ad83-9905ad5ce34e: blacklisted detected during image replay 2019-12-24 02:09:25.707 7f31f5b6f700 -1 rbd::mirror::InstanceReplayer: 0x559dcabd5b80 start_image_replayer: global_image_id=0e614ece-65b1-4b4a-99bd-44dd6235eb70: blacklisted detected during image replay -----------------------------------------------

2 6

Changing failure domain
by Francois Legrand 01 Feb '20

01 Feb '20

Hi, I have a cephfs in production based on 2 pools (data+metadata). Data is in erasure coding with the profile : crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 Metadata is in replicated mode with k=3 The crush rules are as follow : [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "ec_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 5, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] When we installed it, everything was in the same room, but know we splitted our cluster (6 servers but soon 8) in 2 rooms. Thus we updated the crushmap by adding a room layer (with ceph osd crush add-bucket room1 room etc) and move all our servers in the tree to the correct place (ceph osd crush move server1 room=room1 etc...). Now, we would like to change the rules to set a failure domain to room instead of host (to be sure that in case of disaster in one of the rooms we will still have a copy in the other). What is the best strategy to do this ? F.

4 10

Radosgw/Objecter behaviour for homeless session
by Biswajeet Patra 23 Jan '20

23 Jan '20

Hi All, I have a query regarding objecter behaviour for homeless session. In situations when all OSDs containing copies (*let say replication 3*) of an object are down, the objecter assigns a homeless session (OSD=-1) to a client request. This request makes radosgw thread hang indefinitely as the data could never be served because all required OSDs are down. With multiple similar requests, all the radosgw threads gets exhausted and hanged indefinitely waiting for the OSDs to come up. This creates complete service unavailability as no rgw threads are present to process valid requests which could have been directed towards active PGs/OSDs. I think we should have behaviour in objecter or radosgw to terminate request and return early in case of a homeless session. Let me know your thoughts on this. Regards, Biswajeet -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 1

PG lock contention? CephFS metadata pool rebalance
by Stefan Kooman 21 Jan '20

21 Jan '20

Hi, Like I said in an earlier mail to this list, we re-balanced ~ 60% of the CephFS metadata pool to NVMe backed devices. Roughly 422 M objects (1.2 Billion replicated). We have 512 PGs allocated to them. While rebalancing we suffered from quite a few SLOW_OPS. Memory, CPU and device IOPS capacity were not a limiting factor as far as we can see (plenty of them available ... nowhere near max capacity). We saw quite a few slow ops with the following events: "time": "2019-12-19 09:41:02.712010", "event": "reached_pg" }, { "time": "2019-12-19 09:41:02.712014", "event": "waiting for rw locks" }, { "time": "2019-12-19 09:41:02.881939", "event": "reached_pg" ... and this repeated 100's of times taking ~ 30 seconds to complete Does this indicate PG lock contention? If so ... would we need to provide more PGs to the metadata pool to avoid this? The metadata pool is only ~ 166 MiB big ... but with loads of OMAPs ... Most advice on PG planning is concerned with the _amount_ of data ... but the metadata pool (and this might also be true for RGW index pools) seem to be a special case. Thanks for your insights, Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

deep-scrub / backfilling: large amount of SLOW_OPS after upgrade to 13.2.8
by Stefan Kooman 21 Jan '20

21 Jan '20

Hi, After the upgrade to 13.2.8 deep-scrub has a big impact on client IO: loads of SLOW_OPS and high latency. We hardly ever had SLOW_OPS, but since the upgrade the impact is so big that we even have OSDs marking each other out (OSD op thread timeout) multiple times during the scrub window. Plenty of CPU / RAM / IOPS left, hardly any load on these OSD servers. Has there anything changed in this release that can explain this behaviour? Besides this the impact of rebalance is very severe as well. With only the balancer remapping a couple of PGs at a time there are loads of (MDS_)SLOW_OPS. This morning the cephfs metadata pool got rebalanced ... and that triggered a lot of SLOW_OPS. One particular OSD was pegged at 1000% CPU for more than half an hour (not doing that much IO): that's 10 cores going full throttle! After a restart this issue was gone. Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 3

High CPU usage by ceph-mgr in 14.2.5
by Bryan Stillwell 16 Jan '20

16 Jan '20

After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). Attaching to the thread with strace shows a lot of mmap and munmap calls. Here's the distribution after watching it for a few minutes: 48.73% - mmap 49.48% - munmap 1.75% - futex 0.05% - madvise I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this is the only one which has seen the problem (355 OSDs). Perhaps it has something to do with its size? I was suspecting it might have to do with one of the modules misbehaving, so I disabled all of them: # ceph mgr module ls | jq -r '.enabled_modules' [] But that didn't help (I restarted the mgrs after disabling the modules too). I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at me as being the cause of the problem. It only seems to affect the active mgr. If I stop the active mgr the problem moves to one of the other mgrs. Any guesses or tips on what next steps I should take to figure out what's going on? Thanks, Bryan

16 24

why osd's heartbeat partner comes from another root tree?
by opengers 08 Jan '20

08 Jan '20

Hello: According to my understanding, osd's heartbeat partners only come from those osds who assume the same pg See below(# ceph osd tree), osd.10 and osd.0-6 cannot assume the same pg, because osd.10 and osd.0-6 are from different root tree, and pg in my cluster doesn't map across root trees(# ceph osd crush rule dump). so, osd.0-6 cannot become the heartbeat partner of osd.10 But, below is the log on osd.10, It can be seen that the osd.10's heartbeat partner include osd.0/1/2/5, why? thanks for any help > # osd.10 log > 2019-11-20 09:21:50.431799 7fbb369fb700 -1 osd.10 7344 heartbeat_check: no > reply from 10.13.6.162:6806 osd.2 since back 2019-11-20 09:21:19.979712 > front 2019-11-20 09:21:19.979712 (cutoff 2019-11-20 09:21:30.431768) > 2019-11-20 13:15:59.175060 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.162:6806 <http://10.13.6.152:6806> osd.2 since back > 2019-11-20 13:15:38.710424 front 2019-11-20 13:15:38.710424 (cutoff > 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175110 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.160:6803 osd.0 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175118 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.161:6803 osd.1 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-21 02:52:24.656783 7fbb369fb700 -1 osd.10 7374 heartbeat_check: no > reply from 10.13.6.158:6810 osd.5 since back 2019-11-21 02:52:04.557548 > front 2019-11-21 02:52:04.557548 (cutoff 2019-11-21 02:52:04.656781) > # ceph osd tree > -17 3.29095 root ssd-storage > > -25 1.09698 rack rack-ssd-A > > -18 1.09698 host ssd-osd01 > 10 hdd 1.09698 osd.10 up 1.00000 > 1.00000 > -26 1.09698 rack rack-ssd-B > > -19 1.09698 host ssd-osd02 > 11 hdd 1.09698 osd.11 up 1.00000 > 1.00000 > -27 1.09698 rack rack-ssd-C > > -20 1.09698 host ssd-osd03 > 12 hdd 1.09698 osd.12 up 1.00000 > 1.00000 > -1 3.22256 root default > > -3 0.29300 host test-osd01 > > 0 hdd 0.29300 osd.0 up 1.00000 > 1.00000 > -5 0.29300 host test-osd02 > > 1 hdd 0.29300 osd.1 up 0.89999 > 1.00000 > -7 0.29300 host test-osd03 > > 2 hdd 0.29300 osd.2 up 0.79999 > 1.00000 > -9 0.29300 host test-osd04 > > 3 hdd 0.29300 osd.3 up 1.00000 > 1.00000 > -11 0.29300 host test-osd05 > > 4 hdd 0.29300 osd.4 up 1.00000 > 1.00000 > -13 0.29300 host test-osd06 > > 5 hdd 0.29300 osd.5 up 1.00000 > 1.00000 > -15 0.29300 host test-osd07 > > 6 hdd 0.29300 osd.6 up 1.00000 > 1.00000 # ceph osd crush rule dump > > [ > { > "rule_id": 0, > "rule_name": "replicated_rule", > "ruleset": 0, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 1, > "rule_name": "replicated_rule_ssd", > "ruleset": 1, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -17, > "item_name": "ssd-storage" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "rack" > }, > { > "op": "emit" > } > ] > } > ] > # some parameters > "mon_osd_down_out_interval": "600", > "mon_osd_down_out_subtree_limit": "rack", > "mds_debug_subtrees": "false", > "mon_osd_down_out_subtree_limit": "rack", > "mon_osd_reporter_subtree_level": "host",

3 3

MDS failing under load with large cache sizes
by Janek Bevendorff 07 Jan '20

07 Jan '20

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 33

RESEND: Re: PG Balancer Upmap mode not working
by David Zafman 07 Jan '20

07 Jan '20

Please file a tracker with the symptom and examples. Please attach your OSDMap (ceph osd getmap > osdmap.bin). Note that https://github.com/ceph/ceph/pull/31956 has the Nautilus version of improved upmap code. It also changes osdmaptool to match the mgr behavior, so that one can observe the behavior of the upmap balancer offline. Thanks David On 12/8/19 11:04 AM, Philippe D'Anjou wrote: > It's only getting worse after raising PGs now. > > Anything between: > 96 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 97 KiB 13 GiB 4.2 > TiB 53.62 0.76 54 up > > and > > 89 hdd 9.09470 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 88 KiB 21 GiB 1001 > GiB 89.25 1.27 87 up > > How is that possible? I dont know how much more proof I need to > present that there's a bug. > > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

5 20

Balancing PGs across OSDs
by Thomas Schneider 06 Jan '20

06 Jan '20

Hi, in this <https://ceph.io/community/the-first-telemetry-results-are-in/> blog post I find this statement: "So, in our ideal world so far (assuming equal size OSDs), every OSD now has the same number of PGs assigned." My issue is that accross all pools the number of PGs per OSD is not equal. And I conclude that this is causing very unbalanced data placement. As a matter of fact the data stored on my 1.6TB HDD in specific pool "hdb_backup" is in a range starting with osd.228 size: 1.6 usage: 52.61 reweight: 1.00000 and ending with osd.145 size: 1.6 usage: 81.11 reweight: 1.00000 This impacts the amount of data that can be stored in the cluster heavily. Ceph balancer is enabled, but this is not solving this issue. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Therefore I would ask you for suggestions how to work on this unbalanced data distribution. I have attached pastebin for - ceph osd df sorted by usage <https://pastebin.com/QLQHjA9g> - ceph osd df tree <https://pastebin.com/SvhP2hp5> My cluster has multiple crush roots respresenting different disks. In addition I have defined multiple pools, one pool for each disk type: hdd, ssd, nvme. THX

6 23

radosgw - Etags suffixed with #x0e
by Ingo Reimann 03 Jan '20

03 Jan '20

Hi, We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put). I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws. At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types. Any ideas? best regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

2 4

Mimic downgrade (13.2.8 -> 13.2.6) failed assert in combination with bitmap allocator
by Stefan Kooman 27 Dec '19

27 Dec '19

Hi, We have seen several issues (mailed about that earlier to this list) after the upgrade to Mimic 13.2.8. We decided to downgrade the OSD servewrs to 13.2.6 to check if issues disappear. However we ran into issues with that ... We use bluestore allocator since Luminous 12.2.12 to combat latency issues on the OSDs. We also used that succesfully on Mimic 13.2.6. bluestore_allocator = bitmap bluefs_allocator = bitmap When downgrading to 13.2.6 we hit the following assert: 2019-12-27 14:14:16.409 7f2ed2dcce00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 3.5 TiB 2019-12-27 14:14:16.409 7f2ed2dcce00 1 bluefs mount 2019-12-27 14:14:16.413 7f2ed2dcce00 -1 /build/ceph-13.2.6/src/os/bluestore/fastbmap_allocator_impl.h: In function 'void AllocatorLevel02<T>::_mark_allocated(uint64_t, uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7f2ed2dcce00 time 2019-12-27 14:14:16.414793 /build/ceph-13.2.6/src/os/bluestore/fastbmap_allocator_impl.h: 749: FAILED assert(available >= allocated) ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f2eca0a497e] 2: (()+0x2fab07) [0x7f2eca0a4b07] 3: (BitmapAllocator::init_rm_free(unsigned long, unsigned long)+0x44d) [0xc91dbd] 4: (BlueFS::mount()+0x260) [0xc6e6c0] 5: (BlueStore::_open_db(bool, bool)+0x17cd) [0xb8f50d] 6: (BlueStore::_mount(bool, bool)+0x4b7) [0xbbfb77] 7: (OSD::init()+0x295) [0x761fc5] 8: (main()+0x367b) [0x64f23b] 9: (__libc_start_main()+0xf0) [0x7f2ec7c3f830] 10: (_start()+0x29) [0x718929] We upgraded the node back to 13.2.8 again which started without issues. We did do a "downgrade test" on a test cluster ... that cluster did not suffer from this issue. It turned out that the cluster was not using the bitmap allocator ... after enabling the bitmap allocator there on a 13.2.6 node (that has been previviously downgraded but had never run with the bitmap allocator) and restarting the node this came online just fine. However, an upgrade to 13.2.8 with bitmap allocator enabled, and a downgrade again to 13.2.6 would trigger the same assert. Switching back to default (stupid allocator) again would work (initially) for 2 out of 3 OSDs. One would fail right away with rocksdb corruption: 2019-12-27 15:10:50.945 7fc77fbcbe00 20 osd.6 1952 register_pg 2.16 0x990c800 2019-12-27 15:10:50.945 7fc77fbcbe00 10 osd.6:2._attach_pg 2.16 0x990c800 2019-12-27 15:10:50.945 7fc77fbcbe00 10 osd.6 1952 pgid 2.0 coll 2.0_head 2019-12-27 15:10:50.945 7fc77fbcbe00 10 osd.6 1952 _make_pg 2.0 2019-12-27 15:10:50.945 7fc77fbcbe00 5 osd.6 pg_epoch: 1952 pg[2.0(unlocked)] enter Initial 2019-12-27 15:10:50.945 7fc77fbcbe00 20 osd.6 pg_epoch: 1952 pg[2.0(unlocked)] enter NotTrimming 2019-12-27 15:10:50.945 7fc77fbcbe00 -1 abort: Corruption: block checksum mismatch: expected 1122551773, got 2333355710 in db/000397.sst offset 57741 size 4044 2019-12-27 15:10:50.949 7fc77fbcbe00 -1 *** Caught signal (Aborted) ** in thread 7fc77fbcbe00 thread_name:ceph-osd ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) 1: (()+0x11390) [0x7fc775520390] 2: (gsignal()+0x38) [0x7fc774a53428] 3: (abort()+0x16a) [0x7fc774a5502a] 4: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::list*)+0x4a8) [0xbff498] 5: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >*)+0x201) [0xb852e1] 6: (PG::read_info(ObjectStore*, spg_t, coll_t const&, pg_info_t&, PastIntervals&, unsigned char&)+0x16b) [0x7ecc8b] 7: (PG::read_state(ObjectStore*)+0x56) [0x81aff6] 8: (OSD::load_pgs()+0x566) [0x759516] 9: (OSD::init()+0xcd3) [0x762a03] 10: (main()+0x367b) [0x64f23b] 11: (__libc_start_main()+0xf0) [0x7fc774a3e830] 12: (_start()+0x29) [0x718929] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. And after a restart we got rocksdb messages like this: ... 2019-12-27 15:11:32.322 7fd1c0fbbe00 4 rocksdb: [/build/ceph-13.2.6/src/rocksdb/db/version_set.cc:3088] Recovering from manifest file: MANIFEST-000402 ... ... -352> 2019-12-27 15:11:11.598 7fa0f0558e00 -1 abort: Corruption: Bad table magic number: expected 9863518390377041911, found 11124 in db/000397.sst ... After we set osd.6 out ... osd.7 crashed after a while (while backfilling) and would fail to restart again with the following message: 2019-12-27 15:27:10.833 7f1bb4701e00 4 rocksdb: [/build/ceph-13.2.6/src/rocksdb/db/db_impl.cc:252] Shutdown: canceling all background work 2019-12-27 15:27:10.833 7f1bb4701e00 4 rocksdb: [/build/ceph-13.2.6/src/rocksdb/db/db_impl.cc:397] Shutdown complete 2019-12-27 15:27:10.833 7f1bb4701e00 -1 rocksdb: Corruption: CURRENT file does not end with newline 2019-12-27 15:27:10.833 7f1bb4701e00 -1 bluestore(/var/lib/ceph/osd/ceph-7) _open_db erroring opening db: 2019-12-27 15:27:10.833 7f1bb4701e00 1 bluefs umount 2019-12-27 15:27:10.833 7f1bb4701e00 1 stupidalloc 0x0x325aee0 shutdown 2019-12-27 15:27:10.833 7f1bb4701e00 1 bdev(0x380a380 /var/lib/ceph/osd/ceph-7/block) close 2019-12-27 15:27:11.093 7f1bb4701e00 1 bdev(0x380a000 /var/lib/ceph/osd/ceph-7/block) close 2019-12-27 15:27:11.345 7f1bb4701e00 -1 osd.7 0 OSD:init: unable to mount object store 2019-12-27 15:27:11.345 7f1bb4701e00 -1 ESC[0;31m ** ERROR: osd init failed: (5) Input/output errorESC[0m A restart / reboot of the node would not help. For those of you still running 13.2.6 ... I would not recommend upgrading to 13.2.8 (at least not for storage nodes ... mon / mds still seem to work fine). Does bitmap allocator modify the OSD on disk data in some way? Are you supposed to be able to switch between different allocators? Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 0

rgw - ERROR: failed to fetch mdlog info
by Frank R 27 Dec '19

27 Dec '19

I am seeing the following errors on an RGW multisite slave: 1. ERROR: failed to fetch mdlog info 2. failed to fetch local sync status: (5) Input/output error Data seems to be replicating but not metadata. Does anyone have any ideas on what may be wrong? ----- # radosgw-admin sync status realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod) zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod) zone ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1) 2019-12-27 12:29:13.329597 7f71c4ec9dc0 0 meta sync: ERROR: failed to fetch mdlog info metadata sync syncing full sync: 0/64 shards failed to fetch local sync status: (5) Input/output error data sync source: 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 12 shards behind shards: [29,31,33,45,46,48,54,76,87,113,120,127] oldest incremental change not applied: 2019-12-27 12:28:58.0.107159s 23 shards are recovering recovering shards: [1,2,24,26,29,33,35,37,40,41,42,46,48,51,54,66,76,95,100,101,122,123,127] -----

1 0

s3curl putuserpolicy get 405
by 黄明友 26 Dec '19

26 Dec '19

hi,all I follow the https://docs.ceph.com/docs/master/radosgw/STSLite/ ; run s3curl.pl --debug --id admin -- -s -v -X POST "http://localhost:8000/?Action=PutUserPolicy&PolicyName=Policy1&UserName=TES…"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Deny\",\"Action\":\"s3:*\",\"Resource\":\[\"*\"\],\"Condition\":\{\"BoolIfExists\":\{\"sts:authentication\":\"false\"\}\}\},\{\"Effect\":\"Allow\",\"Action\":\"sts:GetSessionToken\",\"Resource\":\"*\",\"Condition\":\{\"BoolIfExists\":\{\"sts:authentication\":\"false\"\}\}\}\]\}&Version=2010-05-08" get error like this, <?xml version="1.0" encoding="UTF-8"?><Error><Code>MethodNotAllowed</Code><RequestId>tx000000000000000001273-005e031dca-e11e-wx6</RequestId><HostId>e11e-wx6-wx</HostId></Error> the ceph verion is ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 黄明友 IT基础架构部经理 V.Photos 云摄影移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy(a)v.photos 官方网址: www.v.photos 上海黄浦区中山东二路88号外滩SOHO3Q F栋 2层北京朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层广州天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区深圳南山区蛇口网谷科技大厦二期A座102网谷双创街 1层成都成华区建设路世贸广场 7层

2 2

Re: Ceph and centos 8
by 林浩 26 Dec '19

26 Dec '19

Hi I can't find el8 rpm package here. http://download.ceph.com/rpm-hammer/ Might I know when the rpm package will be ready for centos8 ?

1 0

rgw logs
by Frank R 25 Dec '19

25 Dec '19

I have about 1TB of data in the pool default.rgw.logs. What logs are stored in this pool and can they be safely deleted? thx Frank

2 1

Ubuntu Bionic arm64 repo missing packages
by Bryan Stillwell 20 Dec '19

20 Dec '19

I was going to try adding an OSD to my home cluster using one of the 4GB Raspberry Pis today, but it appears that the Ubuntu Bionic arm64 repo is missing a bunch of packages: $ sudo grep ^Package: /var/lib/apt/lists/download.ceph.com_debian-nautilus_dists_bionic_main_binary-arm64_Packages Package: ceph-deploy Package: ceph-mgr-dashboard Package: ceph-mgr-diskprediction-cloud Package: ceph-mgr-diskprediction-local Package: ceph-mgr-k8sevents Package: ceph-mgr-rook Package: ceph-mgr-ssh Package: cephfs-shell Package: libcephfs-java Package: python-ceph-argparse Package: python3-ceph-argparse It looks like the packages were built, but for some reason the Packages file is missing them: http://download.ceph.com/debian-nautilus/pool/main/c/ceph/ Could someone look into why the repo isn't finding these packages? Thanks, Bryan

1 1

Copying out bluestore's rocksdb, compact, then put back in - Mimic 13.2.6/13.2.8
by Paul Choi 20 Dec '19

20 Dec '19

Hi, I have a weird situation where an OSD's rocksdb fails to compact, because the OSD became full and the osd-full-ratio was 1.0 (not a good idea, I know). Hitting "bluefs enospc" while compacting: -376> 2019-12-18 15:48:16.492 7f2e0a5ac700 1 bluefs _allocate failed to allocate 0x40da486 on bdev 1, free 0x38b0000; fallback to bdev 2 -376> 2019-12-18 15:48:16.492 7f2e0a5ac700 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne -376> 2019-12-18 15:48:16.492 7f2e0a5ac700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x40da486 -376> 2019-12-18 15:48:16.500 7f2e0a5ac700 -1 /build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uin t64_t, uint64_t)' thread 7f2e0a5ac700 time 2019-12-18 15:48:16.499599 /build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") So my idea is to copy out rocksdb somewhere else (bluefs-export), compact, then copy back in. Is there a way to do this? Mounting bluefs seems to be part of the OSD code, so there's no easy way to do this it seems. Because the OSD died at 100% full, I can't do bluefs-bdev-expand, and repair/fsck fail too. Thanks in advance.

1 1

Prometheus endpoint hanging with 13.2.7 release?
by Paul Choi 20 Dec '19

20 Dec '19

Hello, Anybody seeing the Prometheus endpoint hanging with the new 13.2.7 release? With 13.2.6 the endpoint would respond with a payload of 15MB in less than 10 seconds. Now, if you restart ceph-mgr, the Prometheus endpoint responds quickly for the first run, then successive runs get slower and slower, until it takes several minutes. I have no customization for the mgr module. Except for the Prometheus module, the "status" module and "Zabbix" module are working fine. This is on Ubuntu 16 LTS: ii ceph-mgr 13.2.7-1xenial amd64 manager for the ceph distributed storage system I'd love to know if there's a way to diagnose this issue - I tried upping the debug ms level but that doesn't seem to yield useful information. I don't know if this useful, but "prometheus self-test" is fine too. $ sudo ceph tell mgr.0 prometheus self-test Self-test OK pchoi@pchoi-desktop:~$ ceph mgr module ls { "enabled_modules": [ "prometheus", "status", "zabbix" ], "disabled_modules": [ { "name": "balancer", "can_run": true, "error_string": "" }, { "name": "crash", "can_run": true, "error_string": "" }, { "name": "dashboard", "can_run": true, "error_string": "" }, { "name": "hello", "can_run": true, "error_string": "" }, { "name": "influx", "can_run": false, "error_string": "influxdb python module not found" }, { "name": "iostat", "can_run": true, "error_string": "" }, { "name": "localpool", "can_run": true, "error_string": "" }, { "name": "restful", "can_run": true, "error_string": "" }, { "name": "selftest", "can_run": true, "error_string": "" }, { "name": "smart", "can_run": true, "error_string": "" }, { "name": "telegraf", "can_run": true, "error_string": "" }, { "name": "telemetry", "can_run": true, "error_string": "" } ] } pchoi@pchoi-desktop:~$ ceph mgr services { "prometheus": "http://woodenbox2:9283/" }

2 2

can run more than one rgw multisite realm on one ceph cluster
by 黄明友 20 Dec '19

20 Dec '19

I want run two realm on one ceph cluster, but I found rgw will use only one .rgw.root pool save rgw meta data, so , if run two realm on one ceph cluster, will be error ? 黄明友 IT基础架构部经理 V.Photos 云摄影移动电话: +86 13540630430 客服电话：400 - 806 - 5775 电子邮件: hmy(a)v.photos 官方网址: www.v.photos 上海黄浦区中山东二路88号外滩SOHO3Q F栋 2层北京朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层广州天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区深圳南山区蛇口网谷科技大厦二期A座102网谷双创街 1层成都成华区建设路世贸广场 7层

3 5

PG deep-scrubs ... triggered by backfill?
by Stefan Kooman 20 Dec '19

20 Dec '19

Hi, We have been rebalacing our cluster (CRUSH policy change for cephfs metadata pool). The rebalancing is done ... but now there are PG deep-scrubs going on (mostly on the pool that got almost completely backfilled). We have had the no-(deep)-scrub flags active ... but that does not prevent new (deep-)scrubs from happening. Also disabling them completely on the specific pool did not help ... The oldest deep-scrub was from the 14th of december ... Is there logic coded somewhere that a PG that got backfilled needs to be deep-scrubbed after that? I remember this was a thing on filestore (to check integrity) ... should not be needed anymore with bluestore, right (CRC checking)? All this on Mimic 13.2.8 Thanks, Stefan P.s. I updated https://tracker.ceph.com/issues/11202 -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 0

v14.2.5 Nautilus released
by Abhishek Lekshmanan 20 Dec '19

20 Dec '19

This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release. For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/ Notable Changes --------------- Critical fix: * This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used. New health warnings: * Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded):: ceph crash ls-new To acknowledge a particular crash (or all crashes) and silence the health warning:: ceph crash archive <crash-id> ceph crash archive-all * Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false * Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value:: ceph osd pool set <pool-name> size <num-replicas> The warning can be silenced with:: ceph config set global mon_warn_on_pool_no_redundancy false * A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD. Changes in the telemetry module: * The telemetry module now has a 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: ceph telemetry show ceph telemetry show device # specifically show the device channel If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in: ceph config set mgr mgr/telemetry/channel_crash false ceph telemetry on * The telemetry module now reports more information about CephFS file systems, including: - how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached We have also added: - which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use If you had telemetry enabled, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: ceph telemetry show # see everything ceph telemetry show basic # basic cluster info (including all of the new info) OSD: * A new OSD daemon command, 'dump_recovery_reservations', reveals the recovery locks held (in_progress) and waiting in priority queues. * Another new OSD daemon command, 'dump_scrub_reservations', reveals the scrub reservations that are held for local (primary) and remote (replica) PGs. RGW: * RGW now supports S3 Object Lock set of APIs allowing for a WORM model for storing objects. 6 new APIs have been added put/get bucket object lock, put/get object retention, put/get object legal hold. * RGW now supports List Objects V2 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

9 11

PG-upmap offline optimization is not working as expected
by Thomas Schneider 20 Dec '19

20 Dec '19

Hi, I have tested the PG-upmap offline optimization with 1 of my pools: ssd This pool is unbalanced; here's the ouput of ceph osd df tree before the optimization: root@ld3955:~# ceph osd df tree class ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1532.88501 - 27 TiB 27 GiB 4.4 GiB 816 KiB 23 GiB 27 TiB 0.10 1.00 - root default -46 353.82300 - 1.1 TiB 3.7 GiB 702 MiB 144 KiB 3.0 GiB 1.1 TiB 0.33 3.35 - host ld4257 20 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.35 25 up osd.20 21 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 27 up osd.21 22 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 80 KiB 1024 MiB 370 GiB 0.33 3.35 26 up osd.22 -34 356.22299 - 4.6 TiB 4.9 GiB 936 MiB 228 KiB 4.0 GiB 4.6 TiB 0.10 1.06 - host ld4464 23 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.35 29 up osd.23 24 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 40 KiB 1024 MiB 370 GiB 0.33 3.35 31 up osd.24 25 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 52 KiB 1024 MiB 370 GiB 0.33 3.35 31 up osd.25 26 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 234 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.35 265 up osd.26 -37 356.22299 - 4.6 TiB 4.9 GiB 936 MiB 152 KiB 4.0 GiB 4.6 TiB 0.10 1.06 - host ld4465 27 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 24 up osd.27 28 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 28 up osd.28 29 ssd 0.37099 1.00000 371 GiB 1.2 GiB 234 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.35 22 up osd.29 30 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 234 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.35 258 up osd.30 -3 116.65399 - 4.2 TiB 3.5 GiB 491 MiB 76 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5505 8 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 20 KiB 1024 MiB 3.5 TiB 0.03 0.33 288 up osd.8 9 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 371 GiB 0.31 3.16 28 up osd.9 10 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 32 KiB 1024 MiB 371 GiB 0.31 3.16 31 up osd.10 -7 116.65399 - 4.2 TiB 3.5 GiB 491 MiB 72 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5506 11 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 371 GiB 0.31 3.16 36 up osd.11 12 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 32 KiB 1024 MiB 3.5 TiB 0.03 0.33 260 up osd.12 13 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 16 KiB 1024 MiB 371 GiB 0.31 3.16 28 up osd.13 -10 116.65399 - 4.2 TiB 3.5 GiB 491 MiB 80 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5507 14 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 371 GiB 0.31 3.16 24 up osd.14 15 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 32 KiB 1024 MiB 371 GiB 0.31 3.16 26 up osd.15 16 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 24 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.16 -13 116.65399 - 4.2 TiB 3.5 GiB 490 MiB 64 KiB 3.0 GiB 4.2 TiB 0.08 0.82 - host ld5508 17 ssd 0.37199 1.00000 372 GiB 1.2 GiB 164 MiB 28 KiB 1024 MiB 371 GiB 0.31 3.16 19 up osd.17 18 ssd 0.37199 1.00000 372 GiB 1.2 GiB 163 MiB 8 KiB 1024 MiB 371 GiB 0.31 3.16 24 up osd.18 19 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 164 MiB 28 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.19 TOTAL 27 TiB 27 GiB 4.4 GiB 823 KiB 23 GiB 27 TiB 0.10 MIN/MAX VAR: 0.33/3.35 STDDEV: 0.20 The output of osdmaptool implies many modifications affecting osd.11 and osd.12, means the optimizer wants to shift PGs from 12 to 11. root@ld3955:~# source out_ssd.txt set 66.41 pg_upmap_items mapping to [12->13] set 66.4e pg_upmap_items mapping to [22->20] set 66.7c pg_upmap_items mapping to [28->29] set 66.9f pg_upmap_items mapping to [12->11] set 66.147 pg_upmap_items mapping to [12->11] set 66.1b1 pg_upmap_items mapping to [12->11] set 66.203 pg_upmap_items mapping to [12->11] set 66.257 pg_upmap_items mapping to [28->30] set 66.27d pg_upmap_items mapping to [28->30] set 66.300 pg_upmap_items mapping to [12->11] set 66.354 pg_upmap_items mapping to [28->29] set 66.35b pg_upmap_items mapping to [28->30] set 66.38a pg_upmap_items mapping to [12->11] set 66.3d0 pg_upmap_items mapping to [28->30] However this makes no sense as osd.11 has already more PGs than other OSDs. In fact there's one OSD with least PGs: osd.17 Why is the optimizer not shifting PGs to osd.17? Here's the output of ceph osd df tree after the optimization: root@ld3955:~# ceph osd df tree class ssd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1532.88501 - 27 TiB 27 GiB 3.2 GiB 816 KiB 23 GiB 27 TiB 0.10 1.00 - root default -46 353.82300 - 1.1 TiB 3.6 GiB 502 MiB 144 KiB 3.0 GiB 1.1 TiB 0.33 3.40 - host ld4257 20 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.49 25 up osd.20 21 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.44 27 up osd.21 22 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 80 KiB 1024 MiB 370 GiB 0.31 3.28 26 up osd.22 -34 356.22299 - 4.6 TiB 4.8 GiB 668 MiB 228 KiB 4.0 GiB 4.6 TiB 0.10 1.07 - host ld4464 23 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 44 KiB 1024 MiB 370 GiB 0.33 3.44 29 up osd.23 24 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 40 KiB 1024 MiB 370 GiB 0.31 3.28 31 up osd.24 25 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 52 KiB 1024 MiB 370 GiB 0.33 3.49 31 up osd.25 26 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 167 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.34 265 up osd.26 -37 356.22299 - 4.6 TiB 4.8 GiB 669 MiB 152 KiB 4.0 GiB 4.6 TiB 0.10 1.07 - host ld4465 27 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.31 3.28 24 up osd.27 28 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.33 3.49 28 up osd.28 29 ssd 0.37099 1.00000 371 GiB 1.2 GiB 167 MiB 20 KiB 1024 MiB 370 GiB 0.31 3.28 23 up osd.29 30 ssd 3.48999 1.00000 3.5 TiB 1.2 GiB 167 MiB 92 KiB 1024 MiB 3.5 TiB 0.03 0.36 257 up osd.30 -3 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 76 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5505 8 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 20 KiB 1024 MiB 3.5 TiB 0.03 0.33 288 up osd.8 9 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 371 GiB 0.30 3.13 28 up osd.9 10 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 32 KiB 1024 MiB 371 GiB 0.30 3.13 31 up osd.10 -7 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 72 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5506 11 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 371 GiB 0.30 3.13 41 up osd.11 12 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 32 KiB 1024 MiB 3.5 TiB 0.03 0.33 254 up osd.12 13 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 16 KiB 1024 MiB 371 GiB 0.30 3.13 29 up osd.13 -10 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 80 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5507 14 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 371 GiB 0.30 3.13 24 up osd.14 15 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 32 KiB 1024 MiB 371 GiB 0.30 3.13 26 up osd.15 16 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 24 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.16 -13 116.65399 - 4.2 TiB 3.3 GiB 350 MiB 64 KiB 3.0 GiB 4.2 TiB 0.08 0.81 - host ld5508 17 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 28 KiB 1024 MiB 371 GiB 0.30 3.13 19 up osd.17 18 ssd 0.37199 1.00000 372 GiB 1.1 GiB 117 MiB 8 KiB 1024 MiB 371 GiB 0.30 3.13 24 up osd.18 19 ssd 3.48999 1.00000 3.5 TiB 1.1 GiB 117 MiB 28 KiB 1024 MiB 3.5 TiB 0.03 0.33 259 up osd.19 TOTAL 27 TiB 27 GiB 3.2 GiB 823 KiB 23 GiB 27 TiB 0.10 MIN/MAX VAR: 0.33/3.49 STDDEV: 0.19 Regards Thomas

1 0

ceph radosgw failed to initialize
by dayong tian 20 Dec '19

20 Dec '19

The radosgw can't start normally, the error in log file: ------------------------------- 2019-12-20 14:37:04.058 7fd5b088f700 -1 Initialization timeout, failed to initialize 2019-12-20 14:37:04.304 7fe7148c0780 0 deferred set uid:gid to 167:167 (ceph:ceph) 2019-12-20 14:37:04.304 7fe7148c0780 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process radosgw, pid 3474 2019-12-20 14:37:04.333 7fe6fe47e700 20 reqs_thread_entry: start 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: starting msgr at 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: starting objecter 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: setting wanted keys 2019-12-20 14:37:04.338 7fe7148c0780 1 librados: calling monclient init 2019-12-20 14:37:04.340 7fe7148c0780 1 librados: init done 2019-12-20 14:37:04.340 7fe7148c0780 20 rados->read ofs=0 len=0 2019-12-20 14:37:04.340 7fe7148c0780 10 librados: wait_for_osdmap waiting 2019-12-20 14:37:04.341 7fe7148c0780 10 librados: wait_for_osdmap done waiting 2019-12-20 14:37:04.341 7fe7148c0780 10 librados: read oid=default.realm nspace= 2019-12-20 14:42:04.304 7fe700f0d700 -1 Initialization timeout, failed to initialize ------------------------------- The config I used: ------------------------------- [client.rgw.ceph-test-f01] rgw_host = ceph-test-f01 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-test-f01/keyring rgw_frontends = civetweb port=8099 ------------------------------- The env I set as follows: ------------------------------- # cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) # ceph -v ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) # ceph -s cluster: id: ac018df0-2e71-4c0f-a8a1-0ea29b8a7eb1 health: HEALTH_WARN Reduced data availability: 8 pgs inactive Degraded data redundancy: 8 pgs undersized services: mon: 1 daemons, quorum ceph-test-f01 (age 2h) mgr: ceph-test-f01(active, since 61m) osd: 1 osds: 1 up (since 2h), 1 in (since 2h) data: pools: 2 pools, 40 pgs objects: 1 objects, 1.6 KiB usage: 1.0 GiB used, 98 GiB / 99 GiB avail pgs: 20.000% pgs not active 32 active+clean 8 undersized+peered # ceph osd pool ls .rgw.root panama ------------------------------- I also tried to follow the ceph document to create the default pools such as .default.rgw.control .default.rgw.gc .default.rgw.buckets .default.rgw.buckets.index .default.rgw.buckets.extra .default.log .default.intent-log .default.usage .default.users .default.users.email .default.users.swift .default.users.default.uid The radosgw service still can't start normally, any advices will be appreciated. And all the radosgw-admin utilities hang like: # radosgw-admin zone get 2019-12-20 15:29:03.322 7f18c65876c0 1 librados: starting msgr at 2019-12-20 15:29:03.322 7f18c65876c0 1 librados: starting objecter 2019-12-20 15:29:03.323 7f18c65876c0 1 librados: setting wanted keys 2019-12-20 15:29:03.323 7f18c65876c0 1 librados: calling monclient init 2019-12-20 15:29:03.325 7f18c65876c0 1 librados: init done 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: starting msgr at 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: starting objecter 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: setting wanted keys 2019-12-20 15:29:03.331 7f18c65876c0 1 librados: calling monclient init 2019-12-20 15:29:03.333 7f18c65876c0 1 librados: init done 2019-12-20 15:29:03.334 7f188ffff700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-20 15:29:03.334 7f18c65876c0 20 rados->read ofs=0 len=0 2019-12-20 15:29:03.334 7f188f7fe700 20 reqs_thread_entry: start 2019-12-20 15:29:03.335 7f18c65876c0 10 librados: read oid=default.realm nspace= 2019-12-20 15:29:25.335 7f188ffff700 2 RGWDataChangesLog::ChangesRenewThread: start

1 0

RGW bucket stats extremely slow to respond
by David Monschein 19 Dec '19

19 Dec '19

Hi all! Reaching out again about this issue since I haven't had much luck. We've been seeing some strange behavior with our object storage cluster. While bucket stats (radosgw-admin bucket stats) normally return in a matter of seconds, we frequently observe it taking almost ten minutes, which is not convenient since we use those bucket stats for billing/accounting. Restarting the radosgw process on the RGWs fixes this issue until it crops up again in maybe a few days. Someone mentioned that they think this might have to do with bucket deletions, or more specifically, lifecycle policies to abort incomplete multipart uploads. He mentioned there was an item in the bug tracker for this, but I have not been able to find said bug in the tracker. I have no clue if this is the case or not, but I figured I'd throw it out there to see if anyone else has run into this problem. I have seen many of these messages in my RGW logs: 2019-12-02 13:12:52.882 7faa7018f700 0 abort_bucket_multiparts WARNING : aborted 8553000 incomplete multipart uploads So maybe there is some truth to the aborted multipart uploads causing problems? My cluster has over 200 OSDs, 10 RGWs, about 2200 buckets. Running Nautilus 14.2.5. If anyone has run into this or has any information I'd appreciate it. Merry Christmas & Happy Holidays, - Dave

1 0

Re: Pool Max Avail and Ceph Dashboard Pool Useage on Nautilus giving different percentages
by Stephan Mueller 19 Dec '19

19 Dec '19

Hi, if "MAX AVAIL" displays the wrong data, the bug is just made more visible through the dashboard, as the calculation is correct. To get the right percentage you have to divide the used space through the total, and the total can only consist of two states used and not used space, so both states will be added together to get the total. Or in short: used / (avail + used) Just looked into the C++ code - Max avail will be calculated the following way: avail_res = avail / raw_used_rate ( https://github.com/ceph/ceph/blob/nautilus/src/mon/PGMap.cc#L905) raw_used_rate *= (sum.num_object_copies - sum.num_objects_degraded) / sum.num_object_copies (https://github.com/ceph/ceph/blob/nautilus/src/mon/PGMap.cc#L892) Am Dienstag, den 17.12.2019, 07:07 +0100 schrieb ceph(a)elchaka.de: > I have observed this in the ceph nautilus dashboard too - and Think > it is a Display Bug... but sometimes it Shows tue right values > > > Which nautilus u use? > > > Am 10. Dezember 2019 14:31:05 MEZ schrieb "David Majchrzak, ODERLAND > Webbhotell AB" <david(a)oderland.se>: > > Hi! > > > > While browsing /#/pool in nautilus ceph dashboard I noticed it said > > 93% > > used on the single pool we have (3x replica). > > > > ceph df detal however shows 81% used on the pool and 67% raw > > useage. > > > > # ceph df detail > > RAW STORAGE: > > CLASS SIZE AVAIL USED RAW USED %RAW > > USED > > ssd 478 TiB 153 TiB 324 TiB 325 > > TiB 67.96 > > TOTAL 478 TiB 153 TiB 324 TiB 325 > > TiB 67.96 > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED > > > > MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED > > COMPR UNDER COMPR > > echo 3 108 TiB 29.49M 324 > > TiB 81.61 24 > > TiB N/A N/A 29.49M 0 > > B 0 B I manually calculated the used percentage to get "avail" in your case it seems to be 73 TiB. That means the the total space available for your pool would be 397 TiB. I'm not sure why that is, but it's what the math behind those calculations say. (Found a thread regarding that on the new mailing list (ceph- users(a)ceph.io) -> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NH2LMMX5KVR… ) 0.8161 = used (324) / total => total = 397 Than I looked at the remaining calculations: raw_used_rate *= (sum.num_object_copies - sum.num_objects_degraded) / sum.num_object_copies and avail_res = avail / raw_used_rate First I looked up the init value for "raw_used_rate" for replicated pools. It's their size so we can put in 3 here and for "avail_res" is 24. So I first calculated the final "raw_used_rate" which is 3.042. That means that you have around 4.2% degraded pg's in your pool. > > > > > > I know we're looking at the most full OSD (210PGs, 79% used, 1.17 > > VAR) > > and count max avail from that. But where's the 93% full from in > > dashboard? As said above the calculation is right but the data is wrong... As it uses the real data that can be put in the selected pool, but it uses everywhere else sizes that consider all pool replicas. I created an issue to fix this https://tracker.ceph.com/issues/43384 > > > > My guess is that is comes from calculating: > > > > 1 - Max Avail / (Used + Max avail) = 0.93 > > > > > > Kind Regards, > > > > David Majchrzak > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Hope I could clarify some things and thanks for your feedback :) BTW this problem currently still exists as there wasn't any change to these mentioned lines after the nautilus release. Stephan

1 0

rbd images inaccessible for a longer period of time
by yveskretzschmar＠web.de 19 Dec '19

19 Dec '19

Hi, yesterday I had to power off some vm (proxmox) backed by rbd images for maintenance. After the VMs were off, I tried to create a Snapshot which didn't finish even after half an hour. Because it was maintenance I rebooted all VM nodes an all ceph nodes - nothing changed. Powering on the VM was impossible, kvm exited with timeout. This happened to two of about 15 VM. Two of three Images of one VM still had locks, which I did remove but still unable to power on. I tried to access the Image by mapping it with rbd-nbd, which was unsuccessful and logged this: [ 8601.746971] block nbd0: Connection timed out [ 8601.747648] block nbd0: shutting down sockets [ 8601.747653] block nbd0: Connection timed out [...] [ 8601.750419] block nbd0: Connection timed out [ 8601.750831] print_req_error: 121 callbacks suppressed [ 8601.750832] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 8601.751261] buffer_io_error: 182 callbacks suppressed [ 8601.751262] Buffer I/O error on dev nbd0, logical block 0, async page read [ 8601.751678] blk_update_request: I/O error, dev nbd0, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [...] [ 8601.760283] ldm_validate_partition_table(): Disk read failed. [ 8601.760344] Dev nbd0: unable to read RDB block 0 [ 8601.760985] nbd0: unable to read partition table [ 8601.761282] nbd0: detected capacity change from 0 to 375809638400 [ 8601.761382] ldm_validate_partition_table(): Disk read failed. [ 8601.761461] Dev nbd0: unable to read RDB block 0 [ 8601.762145] nbd0: unable to read partition table The rbd-nbd process kept existing and had to be killed Same thing with qemu-nbd. Exporting the Image via rbd export worked fine, also a rbd copy. Any other operation on the Image (feature dis / enable) took forever so I had to abort it. It seems that every operation leaves a lock on the image. Because it was in the middle of the night, I stopped working on it. Today Morning one of the images was accessible again, the others not. Anybody any hint? Some system information below. Regards, Yves ceph version 14.2.5 (3ce7517553bdd5195b68a6ffaf0bd7f3acad1647) nautilus (stable) Primary Cluster with a backup cluster (rbd mirror) [global] auth client required = none auth cluster required = none auth service required = none auth supported = none cephx_sign_messages = false cephx require signatures = False cluster_network = 172.16.230.0/24 debug asok = 0/0 debug auth = 0/0 debug bdev = 0/0 debug bluefs = 0/0 debug bluestore = 0/0 debug buffer = 0/0 debug civetweb = 0/0 debug client = 0/0 debug compressor = 0/0 debug context = 0/0 debug crush = 0/0 debug crypto = 0/0 debug dpdk = 0/0 debug eventtrace = 0/0 debug filer = 0/0 debug filestore = 0/0 debug finisher = 0/0 debug fuse = 0/0 debug heartbeatmap = 0/0 debug javaclient = 0/0 debug journal = 0/0 debug journaler = 0/0 debug kinetic = 0/0 debug kstore = 0/0 debug leveldb = 0/0 debug lockdep = 0/0 debug mds = 0/0 debug mds balancer = 0/0 debug mds locker = 0/0 debug mds log = 0/0 debug mds log expire = 0/0 debug mds migrator = 0/0 debug memdb = 0/0 debug mgr = 0/0 debug mgrc = 0/0 debug mon = 0/0 debug monc = 0/00 debug ms = 0/0 debug none = 0/0 debug objclass = 0/0 debug objectcacher = 0/0 debug objecter = 0/0 debug optracker = 0/0 debug osd = 0/0 debug paxos = 0/0 debug perfcounter = 0/0 debug rados = 0/0 debug rbd = 0/0 debug rbd mirror = 0/0 debug rbd replay = 0/0 debug refs = 0/0 debug reserver = 0/0 debug rgw = 0/0 debug rocksdb = 0/0 debug striper = 0/0 debug throttle = 0/0 debug timer = 0/0 debug tp = 0/0 debug xio = 0/0 fsid = 27fdf1bb-22a1-4d5e-9729-780cbdcd33fe mon_allow_pool_delete = true mon_host = 172.16.230.142 172.16.230.144 172.16.230.146 mon_osd_down_out_subtree_limit = host osd_backfill_scan_max = 16 osd_backfill_scan_min = 4 osd_deep_scrub_interval = 1209600 osd_journal_size = 5120 osd_max_backfills = 1 osd_max_trimming_pgs = 1 osd_pg_max_concurrent_snap_trims = 1 osd_pool_default_min_size = 2 osd_pool_default_size = 3 osd_recovery_max_active = 1 osd_recovery_max_single_start = 1 osd_recovery_op_priority = 1 osd_recovery_threads = 1 osd_scrub_begin_hour = 19 osd_scrub_chunk_max = 1 osd_scrub_chunk_min = 1 osd_scrub_during_recovery = false osd_scrub_end_hour = 6 osd_scrub_priority = 1 osd_scrub_sleep = 0.5 osd_snap_trim_priority = 1 osd_snap_trim_sleep = 0.005 osd_srub_max_interval = 1209600 public_network = 172.16.230.0/24 max open files = 131072 osd objectstore = bluestore osd op threads = 2 osd crush update on start = true Currently inaccessible image: rbd image 'vm-29009-disk-2': size 200 GiB in 51200 objects order 22 (4 MiB objects) snapshot_count: 2 id: 1abd04da8b9a4d block_name_prefix: rbd_data.1abd04da8b9a4d format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Tue Jul 9 13:07:36 2019 access_timestamp: Thu Dec 19 01:35:34 2019 modify_timestamp: Thu Dec 19 00:19:32 2019 journal: 1abd04da8b9a4d mirroring state: enabled mirroring global id: c71ec81f-18be-4d0b-93ed-0cebe3e619bb mirroring primary: true

2 2

Strange behavior for crush buckets of erasure-profile
by tdados＠hotmail.com 19 Dec '19

19 Dec '19

So I wanted to report a crush rule/ec profile strange behaviour regarding radosgw items which i am not sure if it's a bug or it's supposed to work that way. I am trying to implement the below scenario in my home lab: By default there is a "default" erasure-code-profile with the below settings: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 From the above we see that it uses the root bucket. Now ofcourse you would want to create your own ec-profile with custom algorithm/crush buckets etc Let's say for example we create two new ec profiles. One with specific crush-root = ssd-performance2 and one with the crush-root=default (there are no disks there according to ceph osd tree-> end of page) ceph osd erasure-code-profile set test-ec crush-device-class= crush-failure-domain=host crush-root=ssd-performance2 jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 ceph osd erasure-code-profile set test-ec2 crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=1 plugin=jerasure technique=reed_sol_van w=8 Now let's create the associated crush rules to use these profiles: ceph osd crush rule create-erasure erasure-test-rule1 test-ec ceph osd crush rule create-erasure erasure-test-rule2 test-ec2 Now let's say you have a radosgw server that has started and by default it creates the 5 default radosgwpools(supposed you have uploaded some data as well): default.rgw.buckets.data default.rgw.buckets.index default.rgw.control default.rgw.log default.rgw.meta Now if you grep these pools with ceph osd dump you will see that all of them are using replicated rules but we want to use erasure for the radosgw data pool. So let's migrate the default.rgw.buckets.data pool to a erasure-coded one. 1) We shutdown the radosgw-server so that we don't allow any requests coming in. 2) ceph osd pool rename default.rgw.buckets.data default.rgw.buckets.data-old 3) ceph osd pool create default.rgw.buckets.data 8 8 erasure test-ec erasure-test-rule - > We use the newly created erasure crush rule with the profile we created and use the ssd-performance2 root bucket 4) rados cppool default.rgw.buckets.data-old default.rgw.buckets.data 5) Start radosgw server again At this point i can see the old objects and i can upload new objects in radosgw and everything is working fine. Now i see this strange behavior after i do the below: We set the default.rgw.buckets.data to use the other erasure crush rule (This is using the root bucket=default which doesn't have any disks): ceph osd pool set default.rgw.buckets.data crush_rule erasure-test-rule2 Bug1? You could still browse the data but any attempt to upload/download hangs there with the below log messages: 2019-12-18 17:07:07.037 7f05a1ece700 0 ERROR: client_io->complete_request() returned Input/output error 2019-12-18 17:07:07.037 7f05a1ece700 2 req 712 0.004s s3:list_buckets op status= Monitor nodes don't display anything and seems that new items cannot be saved (which is correct as it doesn't know where to save them) but at least Monitor nodes should display something as a warning or there must be crush check before to see if the rule can be applied? Reverting back the rule to erasure-test-rule works fine again ================================= Bug 2? If you modify the erasure-test-rule profile to use a null crush bucket (like erasure-test-rule2) then this is not being parsed and identified by the crush rule. Seems crush rules skips that part Example: ceph osd erasure-code-profile set test-ec crush-root=default --force At this point nothing happens and radosgw is working fine. Which it shouldn't as it should see that the data cannot be saved anywhere. Unless it keeps the crush root bucket from the crush rules and not from the erasure coded profiles...even if you force apply/change it to the erasure profile like above. ================================= Bug 3? You don't know which rule is using which erasure-code-profile from ceph osd dump. You only see that this pool is using crush rule number 1 but if you dump this crush rule it doesn't mention which erasure-code profile is using, other than which item_name eg = root bucket Even with the telemetry on with latest release and if you do "ceph telemetry show basic" with below you see there is no crush-root being mentioned. So is the crush rule > erasure_code_profile regarding parsing of the crush_root buckets? { "min_size": 2, "erasure_code_profile": { "crush-failure-domain": "host", "k": "2", "technique": "reed_sol_van", "m": "1", "plugin": "jerasure" }, "pg_autoscale_mode": "warn", "pool": 860, "size": 3, "cache_mode": "none", "target_max_objects": 0, "pg_num": 8, "pgp_num": 8, "target_max_bytes": 0, "type": "erasure" } root@ceph-mon01:~# ceph osd crush rule dump erasure-test-rule { "rule_id": 2, "rule_name": "erasure-test-rule", "ruleset": 2, "type": 3, "min_size": 3, "max_size": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "ssd-performance2" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } root@ceph-mon01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -37 0.18398 root really-low -40 0.09799 host ceph-osd01-really-low 11 hdd 0.09799 osd.11 up 1.00000 1.00000 -41 0.04799 host ceph-osd02-really-low 1 hdd 0.01900 osd.1 up 1.00000 1.00000 9 hdd 0.02899 osd.9 up 1.00000 1.00000 -42 0.03799 host ceph-osd03-really-low 6 hdd 0.01900 osd.6 up 1.00000 1.00000 7 hdd 0.01900 osd.7 up 1.00000 1.00000 -23 10.67598 root spinning-rust -20 2.04900 rack rack1 -3 2.04900 host ceph-osd01 3 hdd 0.04900 osd.3 up 0.95001 1.00000 22 hdd 1.00000 osd.22 up 0.90002 1.00000 17 ssd 1.00000 osd.17 up 1.00000 1.00000 -25 3.07799 rack rack2 -5 3.07799 host ceph-osd02 4 hdd 0.04900 osd.4 up 1.00000 1.00000 8 hdd 0.02899 osd.8 up 1.00000 1.00000 23 hdd 1.00000 osd.23 up 1.00000 1.00000 25 hdd 1.00000 osd.25 up 1.00000 1.00000 12 ssd 1.00000 osd.12 up 1.00000 1.00000 -28 3.54900 rack rack3 -7 3.54900 host ceph-osd03 0 hdd 1.00000 osd.0 up 0.90002 1.00000 5 hdd 0.04900 osd.5 up 1.00000 1.00000 30 hdd 0.50000 osd.30 up 1.00000 1.00000 21 ssd 1.00000 osd.21 up 0.95001 1.00000 24 ssd 1.00000 osd.24 up 1.00000 1.00000 -55 2.00000 rack rack4 -49 2.00000 host ceph-osd04 26 hdd 1.00000 osd.26 up 1.00000 1.00000 27 hdd 1.00000 osd.27 up 1.00000 1.00000 -2 9.10799 root ssd-performance2 -32 2.09799 host ceph-osd01-ssd 2 ssd 0.09799 osd.2 up 1.00000 1.00000 13 ssd 1.00000 osd.13 up 1.00000 1.00000 16 ssd 1.00000 osd.16 up 1.00000 1.00000 -31 3.00000 host ceph-osd02-ssd 14 ssd 1.00000 osd.14 up 1.00000 1.00000 18 ssd 1.00000 osd.18 up 1.00000 1.00000 19 ssd 1.00000 osd.19 up 1.00000 1.00000 -9 2.00999 host ceph-osd03-ssd 10 ssd 0.00999 osd.10 up 0.90002 1.00000 15 ssd 1.00000 osd.15 up 1.00000 1.00000 20 ssd 1.00000 osd.20 up 1.00000 1.00000 -52 2.00000 host ceph-osd04-ssd 28 ssd 1.00000 osd.28 up 1.00000 1.00000 29 ssd 1.00000 osd.29 up 1.00000 1.00000 -1 0 root default root@ceph-mon01:~# Thanks, Anastasios

1 0

re-balancing resulting in unexpected availability issues
by steve.nolen＠rstudio.com 18 Dec '19

18 Dec '19

Hi! We've found ourselves in state with our ceph cluster that we haven't seen before, and are looking for a bit of expertise to chime in. We're running a (potentially unusually laid out) moderately large luminous-based ceph cluster in a public cloud, with 234*8TB OSDs, with a single osd per cloud instance. Here's a snippet of our ceph status: services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon4 mgr: ceph-mon2(active), standbys: ceph-mon1, ceph-mon4 osd: 234 osds: 234 up, 231 in data: pools: 5 pools, 7968 pgs objects: 136.35M objects, 382TiB usage: 1.09PiB used, 713TiB / 1.79PiB avail pgs: 7924 active+clean 44 active+clean+scrubbing+deep Our layout is spread across 6 availability zones (with the majority of osds in three, us-east-1a,us-east-1b and us-east-1e). We've recently decided that a spread across six azs is unnecessary and potentially a detriment to performance, so we are working towards shifting our workload out of 3 of the 6 azs, so that we are evenly placed in 3 azs. Relevant Recent events: 1. We expanded by 24 osds to handle additional capacity needs as well as to provide the capacity necessary to remove all osds in us-east-1f. 2. After the re-balance related to #1 finished we expanded by an additional 12 osds as a follow-on for the #1 change. 3. On the same day as #2, we also issued `ceph crush move` commands to move the location of oldest 20 osds which had previously not been configured with a "datacenter" bucket denoting their availability zone. The re-balancing related to #3 caused quite a change in our cluster, resulting in hours with degraded pgs, and waves of "slow requests" from many of the osds as data shifted. Three of the osds which had their location moved are also wildly more full than the other osds (being in a nearfull, >85% utilized state where the other 231 osds are in the range of 58-63% utilized). A `ceph balancer optimize` plan shows no changes to be made by the balancer to rectify this. Because of their nearfull status, we have marked those three osds as "out". During the resulting re-balancing we experienced more "slow requests" piling up, with some pgs dipping into a peering or activating state (which obviously causes some user-visible trouble). Where we are headed: * We have yet to set the crush map to use a replicated_datacenter rule now that our osds conform to these buckets. * We need to take action to remove/replace the three osds which were over-utilized. * We need to remove the us-east-1f osds that provoked the event in #1 (of which there are 16). What I'm hoping for a bit of direction on: * A recommendation on the order of these operations that would result in the least user-visible time. * Some hints or thoughts on why we are seeing flapping pg availability since the change in #3. * General thoughts on our layout. It has been rather useful for unbounded rapid growth but it seems very likely the sheer count of osds/instances is optimal at this point. Obviously I'm happy to provide any additional information that might be helpful that I've overlooked. Thanks so much for looking! Steve

1 0

Use Wireshark to analysis ceph network package
by Xu Chen 18 Dec '19

18 Dec '19

Hi guys, I want to use tcpdump and Wireshark to capture and analysis packages between clients and ceph cluster. But the protocol only shows tcp, no ceph, so I can not read the data between client and cluster. The wireshark version is 3.07. Hope your help. Thank you.

1 0

list CephFS snapshots
by Lars Täuber 18 Dec '19

18 Dec '19

Hi! Is there a mean to list all snapshots existing in a (subdir of) Cephfs? I can't use the find dommand to look for the ".snap" dirs. I'd like to remove certain (or all) snapshots within a CephFS. But how do I find them? Thanks, Lars

4 9

what's meaning of "cache_hit_rate": 0.000000 in "ceph daemon mds.<x> dump loads" output?
by Sang, Oliver 18 Dec '19

18 Dec '19

hi all, seek your help for this. we are using luminous 12.2.12 and we enabled 3 active mds. when I running "ceph daemon mds.<x> dump loads" on any active mds, I always saw such like below "mds_load": { "mds.0": { "request_rate": 526.045993, "cache_hit_rate": 0.000000, ... "mds.1": { "request_rate": 169.845956, "cache_hit_rate": 0.000000, ... "mds.2": { "request_rate": 300.511478, "cache_hit_rate": 0.000000, don't understand what's this 'cache_hit_rate' and why it seems always 0? we actually set a bigger mds_cache_memory_limit than default such like 16G in /etc/ceph/ceph.conf. are they related? I checked https://docs.ceph.com/docs/luminous/cephfs/mds-config-ref/ , but didn't get further clue from the description of "mds cache *" settings.

1 0

Ceph and centos 8
by fleg＠lpnhe.in2p3.fr 17 Dec '19

17 Dec '19

Hi, We have a ceph+cephfs cluster runing nautilus version 14.2.4 We have debian buster/ubuntu bionic clients mounting cephfs in kernel mode without problems. We now want to mount cephfs from our new centos 8 clients. Unfortunately, ceph-common is needed but there are no packages available for el8 (only el7). And no way to install the el7 packages on centos 8 (missing deps). Thus, despite the fact that centos 8 have a 4.18 kernel (required to use quota, snapshots etc...), it seems impossible to mount in kernel mode (good perfs) and we still have to use the so slow fuse mode. Is it possible to workaround this problem ? Or when is it planned to provides (even as beta) the ceph packages for centos 8 ? Thanks.

4 5

MGR log reports error related to Ceph Dashboard: SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727)
by Thomas Schneider 17 Dec '19

17 Dec '19

Hello, I have enabled MGR module "Dashboard" and created a self-signed cert: root@ld5506:~# ceph dashboard create-self-signed-cert Self-signed certificate created Checking the MGR log I get multiple errors for any action in the dashboard with Ceph version 14.2.4.1 (596a387fb278758406deabf997735a1f706660c9) nautilus (stable): 2019-12-17 08:32:30.203 7f70942eb700 0 mgr[dashboard] [17/Dec/2019:08:32:30] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start self.tick() File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap server_side=True) File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727) What is causing this error? How can I fix it? THX

1 0

bluestore worries
by Frank R 16 Dec '19

16 Dec '19

Hi all, I am thinking about converting a Filestore cluster to Bluestore. The OSD nodes have 16X4TB 7200 SATA OSDs with NVME write journals. The NVME drives should be large enough to house ~30G DB/WAL OSDs. I am worried that I will see a significant performance hit when the deferred writes to the NVME journals are eliminated with Bluestore. Has anyone converted a similar setup to Bluestore? If so, what was the performance impact. thx Frank

3 2

Re: ceph osd pool ls detail 'removed_snaps' on empty pool?
by Gregory Farnum 16 Dec '19

16 Dec '19

On Mon, Dec 16, 2019 at 11:34 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > Hi Gregory, > > I saw ceph -s showing 'snaptrim'(?), but I have still have these > 'removed_snaps' listed on this pool (also on other pools, I don't > remember creating/deleting them) A 'ceph tell mds.c scrub start /test/ > recursive repair' did not remove those. Can/should/how I remove these? > > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] ceph osd pool ls detail 'removed_snaps' on empty > pool? > > > I have removed_snaps listed on pools that I am not using. They are > mostly for doing some performance testing, so I cannot imagine ever > creating snapshots in them. > > > pool 33 'fs_data.ssd' replicated size 3 min_size 1 crush_rule 5 > object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn > last_change 68413 lfor 0/0/65853 flags hashpspool,selfmanaged_snaps > stripe_width 0 application cephfs > removed_snaps > [567~1,56b~1,56f~1,57f~1,583~1,587~1,58b~1,58f~1,591~1,593~1,595~1,597~1 > ,599~1,59b~1,59d~1,59f~1,5a1~1,5a3~1,5a5~1,5a7~1, These are RADOS snapshots, so CephFS may have removed them but the OSDs still need to trim all the data; cephfs commands won't do anything to or with them. Once the OSD have removed the snapshot (it can take a while, depending on settings and other cluster activity) they'll report it back to the monitor, and it will remove them from the list of removed snaps. (It may not bother removing them if there aren't enough other live snapshots, I forget.) Either way, this list is basically harmless so you shouldn't worry about it. > > > [@ ]# rados df| egrep '^POOL|fs_data.ssd' > POOL_NAME USED OBJECTS CLONES COPIES > MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS > WR USED COMPR UNDER COMPR > fs_data.ssd 0 B 0 0 0 > > 0 0 0 0 0 B 0 0 B 0 > B 0 B > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

How safe is k=2, m=1, min_size=2?
by Dan van der Ster 16 Dec '19

16 Dec '19

Hi all, We have said in the past that an EC pool should have min_size=k+1, for the same reasons that a replica 3 pool needs min_size=2. And we've heard several stories about replica 3, min_size=1 leading to incomplete PGs. Taking a quick poll -- did anyone ever suffer an outage on a pool with k=2, m=1, min_size=2? Thanks! Dan

2 1

ceph osd pool ls detail 'removed_snaps' on empty pool?
by Marc Roos 16 Dec '19

16 Dec '19

I have removed_snaps listed on pools that I am not using. They are mostly for doing some performance testing, so I cannot imagine ever creating snapshots in them. pool 33 'fs_data.ssd' replicated size 3 min_size 1 crush_rule 5 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 68413 lfor 0/0/65853 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs removed_snaps [567~1,56b~1,56f~1,57f~1,583~1,587~1,58b~1,58f~1,591~1,593~1,595~1,597~1 ,599~1,59b~1,59d~1,59f~1,5a1~1,5a3~1,5a5~1,5a7~1, [@ ]# rados df| egrep '^POOL|fs_data.ssd' POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR fs_data.ssd 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

1 1

v13.2.7 mimic released
by Sage Weil 16 Dec '19

16 Dec '19

This is the seventh bugfix release of the Mimic v13.2.x long term stable release series. We recommend all Mimic users upgrade. For the full release notes, see https://ceph.io/releases/v13-2-7-mimic-released/ Notable Changes MDS: - Cache trimming is now throttled. Dropping the MDS cache via the “ceph tell mds.<foo> cache drop” command or large reductions in the cache size will no longer cause service unavailability. - Behavior with recalling caps has been significantly improved to not attempt recalling too many caps at once, leading to instability. MDS with a large cache (64GB+) should be more stable. - MDS now provides a config option “mds_max_caps_per_client” (default: 1M) to limit the number of caps a client session may hold. Long running client sessions with a large number of caps have been a source of instability in the MDS when all of these caps need to be processed during certain session events. It is recommended to not unnecessarily increase this value. - The “mds_recall_state_timeout” config parameter has been removed. Late client recall warnings are now generated based on the number of caps the MDS has recalled which have not been released. The new config parameters “mds_recall_warning_threshold” (default: 32K) and “mds_recall_warning_decay_rate” (default: 60s) set the threshold for this warning. - The “cache drop” admin socket command has been removed. The “ceph tell mds.X cache drop” remains. OSD: - A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option “mon_warn_on_slow_ping_ratio” specifies a percentage of “osd_heartbeat_grace” to determine the threshold. A value of zero disables the warning. A new configuration option “mon_warn_on_slow_ping_time”, specified in milliseconds, overrides the computed value, causing a warning when OSD heartbeat pings take longer than the specified amount. A new admin command “ceph daemon mgr.# dump_osd_network [threshold]” lists all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. A new admin command ceph daemon osd.# dump_osd_network [threshold]” does the same but only including heartbeats initiated by the specified OSD. - The default value of the “osd_deep_scrub_large_omap_object_key_threshold” parameter has been lowered to detect an object with large number of omap keys more easily. RGW: - radosgw-admin introduces two subcommands that allow the managing of expire-stale objects that might be left behind after a bucket reshard in earlier versions of RGW. One subcommand lists such objects and the other deletes them. Read the troubleshooting section of the dynamic resharding docs for details.

4 7

any way to read magic number like #1018a1b3c14?
by Sang, Oliver 16 Dec '19

16 Dec '19

we are using cephfs. in /sys/kernel/debug/ceph/*/mdsc sometimes we saw something like - 2015909 mds0 setfilelock #1018a1b3c14 I know this #1018a1b3c14 is a file or a folder, but how to know what it really is in cephfs tree? Thanks!

1 0

Re: ceph-volume sizing osds
by Jan Fajerski 14 Dec '19

14 Dec '19

On Fri, Dec 13, 2019 at 11:49:50PM +0100, Oscar Segarra wrote: > Hi, > I have recently started working with Ceph Nautilus release and I have > realized that you have to start working with LVM to create OSD instead > of the "old fashioned" ceph-disk. > In terms of performance and best practices, as far as I must use LVM I > can create volume groups that joins or extends two or more physical > disks. In this scenario (many disks for server) where ceph-volume is > manatory, It still remains the rule of one OSD for each physical > device? or I can reduce the number of OSDs? That is hard to answer generically. ceph-volume will not create multi-disk volumes. One disk, one OSD is still the recommendation afaik. However there are certainly scenarios where multi-disk OSDs are sensible. For example if you have a machine that has _a lot_ of disks attached but lacks CPU and RAM to run as many OSDs as disks, it can make sense to deploy one OSD on multiple disks. > Thanks in advance, > Óscar >_______________________________________________ >ceph-users mailing list >ceph-users(a)lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

Re: Can't create new OSD
by Gregory Farnum 13 Dec '19

13 Dec '19

[ Re-adding users list ] On Thu, Dec 12, 2019 at 10:12 AM Rodrigo Severo - Fábrica <rodrigo(a)fabricadeideias.com> wrote: > > Em qui., 12 de dez. de 2019 às 12:55, Gregory Farnum > <gfarnum(a)redhat.com> escreveu: > > > > On Wed, Dec 11, 2019 at 12:04 PM Rodrigo Severo - Fábrica <rodrigo(a)fabricadeideias.com> wrote: > >> > >> Hi, > >> > >> > >> Trying to create a new OSD following the instructions available at > >> https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/ > >> > >> On step 3 I'm instructed to run "ceph-osd -i {osd-num} --mkfs > >> --mkkey". Unfortunately it doesn't work: > >> > >> # ceph-osd -i 3 --mkfs --mkkey > >> 2019-12-11 16:59:58.257 7fac4597fc00 -1 auth: unable to find a keyring > >> on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > >> 2019-12-11 16:59:58.257 7fac4597fc00 -1 AuthRegistry(0x55ad976ea140) > >> no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > >> 2019-12-11 16:59:58.261 7fac4597fc00 -1 auth: unable to find a keyring > >> on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > >> 2019-12-11 16:59:58.261 7fac4597fc00 -1 AuthRegistry(0x7fffac4075e8) > >> no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > >> failed to fetch mon config (--no-mon-config to skip) > > I'm full of questions. > > > This is the important bit. Since you’re running the command without a way to access the Ceph monitors, > > Should I provide a way do access the Ceph monitors? I was just > following the docs and there is no mention to it in the above page. > > Anyway I tried to do it but the results were exactly the same: > > # ceph-osd -m 192.168.109.233:3300 -i 3 --mkfs --mkkey > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 AuthRegistry(0x564545484140) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:56:18.408 7f392c0a1c00 -1 AuthRegistry(0x7ffccdef0958) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > I also tried in the 6789 port and without providing port at all. > Always the same results. Tried also on the other monitors. Always the > same results. > > > it can’t use the cluster default configuration. You need to move the client.admin keyring onto the node (or another one, if you configure the ceph.conf or pass in the name on each command invocation). I haven't been through the docs in a while so maybe it's missing or lost but it should describe that somewhere. This is necessary whenever you are making changes to the cluster, such as adding an OSD — without a keyring, the cluster has no idea if the user making the change is authorized! > > Shouldn't it automatically read /etc/ceph/ceph.conf and figure everything out? > > I also tried explicitly setting the conf path but, again, same results: It's reading the ceph.conf but still needs cluster access, which you haven't given it. > > # ceph-osd -i 3 --mkfs --mkkey -c /etc/ceph/ceph.conf > 2019-12-12 14:57:35.740 7fee29b61c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:35.740 7fee29b61c00 -1 AuthRegistry(0x56457c784140) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > 2019-12-12 14:57:35.740 7fee29b61c00 -1 auth: unable to find a keyring > on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:35.740 7fee29b61c00 -1 AuthRegistry(0x7ffc6eccb0b8) > no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx > failed to fetch mon config (--no-mon-config to skip) > > > You can pass the given option to not worry about that, > > Tried that. Apparently it worked, despite several disturbing messages: > > # ceph-osd -i 3 --mkfs --mkkey --no-mon-config > 2019-12-12 14:57:50.256 7f31d77b2c00 -1 auth: error reading file: > /var/lib/ceph/osd/ceph-3/keyring: can't open > /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 created new key in keyring > /var/lib/ceph/osd/ceph-3/keyring > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.260 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to > open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory > 2019-12-12 14:57:50.629 7f31d77b2c00 -1 > bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid > # ll /var/lib/ceph/osd/ceph-3 > total 172 > drwxr-xr-x 2 root root 4096 Dec 12 14:57 ./ > drwxr-xr-x 4 ceph ceph 4096 Dec 11 16:19 ../ > -rw-r----- 1 root root 10737418240 Dec 12 14:57 block > -rw------- 1 root root 2 Dec 12 14:57 bluefs > -rw------- 1 root root 37 Dec 12 14:57 ceph_fsid > -rw-r----- 1 root root 37 Dec 12 14:57 fsid > -rw------- 1 root root 56 Dec 12 14:57 keyring > -rw------- 1 root root 8 Dec 12 14:57 kv_backend > -rw------- 1 root root 21 Dec 12 14:57 magic > -rw------- 1 root root 4 Dec 12 14:57 mkfs_done > -rw------- 1 root root 6 Dec 12 14:57 ready > -rw------- 1 root root 10 Dec 12 14:57 type > -rw------- 1 root root 2 Dec 12 14:57 whoami > > Is it how it should be? With all these "failed to open" messages? Yeah, those are BlueStore et al attempting various opens that are allowed to fail. I agree they're not great to have show up as standard. > > > or you can run the command with a user key ring that’s allowed to read the monitor configs. (You’ll need that key access next to add the OSD’s key to the cluster, anyway!) > > Haven't tried that. How is it done? I can't find any option to provide > a user key ring in ceph-osd's man page. It's not ceph-osd specific, just anywhere in ceph. -n client.[name], -k [keyring_file] etc > > > Many thanks for your help. > > Rodrigo >

1 0

v13.2.8 Mimic released
by Abhishek Lekshmanan 13 Dec '19

13 Dec '19

This is the eighth backport release in the Ceph Mimic stable release series. Its sole purpose is to fix a regression that found its way into the previous release. Notable Changes --------------- * Due to a missed backport, clusters in the process of being upgraded from 13.2.6 to 13.2.7 might suffer an OSD crash in build_incremental_map_msg. This regression was reported in https://tracker.ceph.com/issues/43106 and is fixed in 13.2.8 (this release). Users of 13.2.6 can upgrade to 13.2.8 directly - i.e., skip 13.2.7 - to avoid this. Changelog --------- * osd: fix sending incremental map messages (issue#43106 pr#32000, Sage Weil) * tests: added missing point release versions (pr#32087, Yuri Weinstein) * tests: rgw: add missing force-branch: ceph-mimic for swift tasks (pr#32033, Casey Bodley) For a blog with links to PRs and issues please check out https://ceph.io/releases/v13-2-8-mimic-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-13.2.8.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

1 0

Re: RESEND: Re: PG Balancer Upmap mode not working
by David Zafman 12 Dec '19

12 Dec '19

Phillippe, Maybe you can try the "crush-compat" balancer mode instead of "upmap" until the new code is released. David On 12/11/19 9:36 PM, Philippe D'Anjou wrote: > > Hi, > I see your code balanced my ssdpool to about 146 each. > I can confirm this did NOT happen. > The state it ended up in is: > > 0 ssd 3.49219 1.00000 3.5 TiB 797 GiB 757 GiB 36 GiB 3.9 GiB 2.7 > TiB 22.30 0.31 147 up > 1 ssd 3.49219 1.00000 3.5 TiB 803 GiB 751 GiB 49 GiB 3.7 GiB 2.7 > TiB 22.47 0.31 146 up > 2 ssd 3.49219 1.00000 3.5 TiB 818 GiB 764 GiB 51 GiB 3.8 GiB 2.7 > TiB 22.89 0.32 150 up > 3 ssd 3.49219 1.00000 3.5 TiB 794 GiB 757 GiB 34 GiB 3.2 GiB 2.7 > TiB 22.21 0.31 146 up > 4 ssd 3.49219 1.00000 3.5 TiB 837 GiB 798 GiB 34 GiB 4.4 GiB 2.7 > TiB 23.39 0.32 156 up > 6 ssd 3.49219 1.00000 3.5 TiB 790 GiB 751 GiB 35 GiB 3.6 GiB 2.7 > TiB 22.09 0.31 146 up > 8 ssd 3.49219 1.00000 3.5 TiB 874 GiB 831 GiB 40 GiB 3.5 GiB 2.6 > TiB 24.44 0.34 156 up > 10 ssd 3.49219 1.00000 3.5 TiB 807 GiB 761 GiB 43 GiB 3.4 GiB 2.7 > TiB 22.58 0.31 146 up > 5 ssd 3.49219 1.00000 3.5 TiB 744 GiB 708 GiB 32 GiB 4.2 GiB 2.8 > TiB 20.81 0.29 141 up > 7 ssd 3.49219 1.00000 3.5 TiB 732 GiB 690 GiB 39 GiB 3.2 GiB 2.8 > TiB 20.48 0.28 136 up > 9 ssd 3.49219 1.00000 3.5 TiB 702 GiB 657 GiB 42 GiB 3.9 GiB 2.8 > TiB 19.64 0.27 131 up > 11 ssd 3.49219 1.00000 3.5 TiB 805 GiB 781 GiB 22 GiB 2.3 GiB 2.7 > TiB 22.50 0.31 138 up > 101 ssd 3.49219 1.00000 3.5 TiB 835 GiB 793 GiB 38 GiB 3.7 GiB 2.7 > TiB 23.36 0.32 146 up > 103 ssd 3.49219 1.00000 3.5 TiB 846 GiB 803 GiB 40 GiB 3.3 GiB 2.7 > TiB 23.67 0.33 150 up > 105 ssd 3.49219 1.00000 3.5 TiB 800 GiB 762 GiB 36 GiB 2.5 GiB 2.7 > TiB 22.38 0.31 148 up > 107 ssd 3.49219 1.00000 3.5 TiB 843 GiB 790 GiB 49 GiB 3.4 GiB 2.7 > TiB 23.58 0.33 147 up > 100 ssd 3.49219 1.00000 3.5 TiB 804 GiB 753 GiB 48 GiB 2.6 GiB 2.7 > TiB 22.47 0.31 144 up > 102 ssd 3.49219 1.00000 3.5 TiB 752 GiB 737 GiB 13 GiB 2.4 GiB 2.8 > TiB 21.02 0.29 141 up > 104 ssd 3.49219 1.00000 3.5 TiB 805 GiB 771 GiB 31 GiB 2.8 GiB 2.7 > TiB 22.50 0.31 144 up > 106 ssd 3.49219 1.00000 3.5 TiB 793 GiB 724 GiB 66 GiB 2.9 GiB 2.7 > TiB 22.17 0.31 143 up > 108 ssd 3.49219 1.00000 3.5 TiB 816 GiB 778 GiB 36 GiB 2.7 GiB 2.7 > TiB 22.83 0.32 156 up > 109 ssd 3.49219 1.00000 3.5 TiB 811 GiB 763 GiB 45 GiB 2.8 GiB 2.7 > TiB 22.68 0.31 146 up > 110 ssd 3.49219 1.00000 3.5 TiB 863 GiB 832 GiB 28 GiB 2.5 GiB 2.6 > TiB 24.13 0.33 154 up > 111 ssd 3.49219 1.00000 3.5 TiB 784 GiB 737 GiB 45 GiB 2.7 GiB 2.7 > TiB 21.92 0.30 146 up > > > It did not try to balance any further. Someone said he had the same issue. > I am pretty sure it will also not balance out the HDDs as neatly as > you got it there. There is definitely an issue somewhere, so far 3 > people telling the same story. I never had this issue under Luminous > but im fighting with it since 4 months on 2 clusters. One got upgraded > to Nautilus and the other one (the one the pastes are from) is a fresh > 14.2.4 one. > > Any ideas on that? > > Thanks > Am Donnerstag, 12. Dezember 2019, 02:09:33 OEZ hat David Zafman > <dzafman(a)redhat.com> Folgendes geschrieben: > > > > Philippe, > > I have a master branch version of the code to test. The nautilus > backport https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 >should be the same. > > Using your OSDMap, the code in master branch and some additional changes > to osdmaptool I was able to balance your cluster. The osdmaptool > changes simulate the mgr active balancer behavior. It never took no > more than 0.13991 seconds to calculate more upmaps per round. And that's > on a virtual machine used for development. It took 35 rounds with 10 > maximum upmaps per crush rule set of pools per round. With the default > 1 minute sleeps inside the mgr it would take 35 minutes. Obviously, > recovery/backfill has to finish before the cluster settles into the new > configuration. It needed 397 additional upmaps and removed 8. > > Because all pools for a given crush rule are balanced together you can > see that this is more balanced than Rich's configuration uising Luminous. > > This balancer code is subject to change before final release of the next > Nautilus point release. > > > Final layout: > > osd.0 pgs 146 > osd.1 pgs 146 > osd.2 pgs 146 > osd.3 pgs 146 > osd.4 pgs 146 > osd.5 pgs 146 > osd.6 pgs 146 > osd.7 pgs 146 > osd.8 pgs 146 > osd.9 pgs 146 > osd.10 pgs 146 > osd.11 pgs 146 > osd.12 pgs 74 > osd.13 pgs 74 > osd.14 pgs 73 > osd.15 pgs 74 > osd.16 pgs 74 > osd.17 pgs 74 > osd.18 pgs 73 > osd.19 pgs 74 > osd.20 pgs 73 > osd.21 pgs 73 > osd.22 pgs 74 > osd.23 pgs 73 > osd.24 pgs 73 > osd.25 pgs 75 > osd.26 pgs 74 > osd.27 pgs 74 > osd.28 pgs 73 > osd.29 pgs 73 > osd.30 pgs 73 > osd.31 pgs 73 > osd.32 pgs 74 > osd.33 pgs 73 > osd.34 pgs 73 > osd.35 pgs 74 > osd.36 pgs 74 > osd.37 pgs 74 > osd.38 pgs 74 > osd.39 pgs 74 > osd.40 pgs 73 > osd.41 pgs 73 > osd.42 pgs 73 > osd.43 pgs 73 > osd.44 pgs 74 > osd.45 pgs 73 > osd.46 pgs 73 > osd.47 pgs 73 > osd.48 pgs 73 > osd.49 pgs 73 > osd.50 pgs 73 > osd.51 pgs 73 > osd.52 pgs 75 > osd.53 pgs 59 > osd.54 pgs 74 > osd.55 pgs 74 > osd.56 pgs 74 > osd.57 pgs 73 > osd.58 pgs 74 > osd.59 pgs 74 > osd.60 pgs 74 > osd.61 pgs 74 > osd.62 pgs 73 > osd.63 pgs 74 > osd.64 pgs 73 > osd.65 pgs 74 > osd.66 pgs 74 > osd.67 pgs 74 > osd.68 pgs 73 > osd.69 pgs 74 > osd.70 pgs 73 > osd.71 pgs 73 > osd.72 pgs 73 > osd.73 pgs 73 > osd.74 pgs 73 > osd.75 pgs 73 > osd.76 pgs 73 > osd.77 pgs 73 > osd.78 pgs 73 > > osd.79 pgs 73 > osd.80 pgs 73 > osd.81 pgs 73 > osd.82 pgs 73 > osd.83 pgs 73 > osd.84 pgs 73 > osd.85 pgs 73 > osd.86 pgs 73 > osd.87 pgs 73 > osd.88 pgs 73 > osd.89 pgs 73 > osd.90 pgs 73 > osd.91 pgs 73 > osd.92 pgs 73 > osd.93 pgs 73 > osd.94 pgs 73 > osd.95 pgs 73 > osd.96 pgs 73 > osd.97 pgs 73 > osd.98 pgs 73 > osd.99 pgs 73 > osd.100 pgs 146 > osd.101 pgs 146 > osd.102 pgs 146 > osd.103 pgs 146 > osd.104 pgs 146 > osd.105 pgs 146 > osd.106 pgs 146 > osd.107 pgs 146 > osd.108 pgs 146 > osd.109 pgs 146 > osd.110 pgs 146 > osd.111 pgs 146 > osd.112 pgs 73 > osd.113 pgs 73 > osd.114 pgs 73 > osd.115 pgs 73 > osd.116 pgs 73 > osd.117 pgs 73 > osd.118 pgs 73 > osd.119 pgs 73 > osd.120 pgs 73 > osd.121 pgs 73 > osd.122 pgs 73 > osd.123 pgs 73 > osd.124 pgs 73 > osd.125 pgs 73 > osd.126 pgs 73 > osd.127 pgs 74 > osd.128 pgs 73 > osd.129 pgs 73 > osd.130 pgs 73 > osd.131 pgs 73 > osd.132 pgs 73 > osd.133 pgs 73 > osd.134 pgs 73 > osd.135 pgs 73 > > > David > > On 12/10/19 9:59 PM, Philippe D'Anjou wrote: > > Given I was told its an issue of too low PGs I am raising and testing > > this, although my SSDs which have about 150 each also are not well > > distributed. > > I attached my OSDmap, I'd appreciate if you could run your test on it > > like you did with the other guy, so I know if this will ever > > distribute equally or not.. > > > > If you're busy I understand that too, then ignore this. > > > > Thanks in either case. Just have been dealing with this since months > > now and it gets frustrating. > > > > best regards > > > > Am Dienstag, 10. Dezember 2019, 03:53:17 OEZ hat David Zafman > > <dzafman(a)redhat.com <mailto:dzafman@redhat.com>> Folgendes geschrieben: > > > > > > > > Please file a tracker with the symptom and examples. Please attach your > > OSDMap (ceph osd getmap > osdmap.bin). > > > > Note that https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 > > > <https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 >>has the Nautilus > > version of improved upmap code. It also changes osdmaptool to match the > > mgr behavior, so that one can observe the behavior of the upmap balancer > > offline. > > > > Thanks > > > > David > > > > On 12/8/19 11:04 AM, Philippe D'Anjou wrote: > > > It's only getting worse after raising PGs now. > > > > > > Anything between: > > > 96 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 97 KiB 13 GiB 4.2 > > > TiB 53.62 0.76 54 up > > > > > > and > > > > > > 89 hdd 9.09470 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 88 KiB 21 GiB 1001 > > > GiB 89.25 1.27 87 up > > > > > > How is that possible? I dont know how much more proof I need to > > > present that there's a bug. > > > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> > <mailto:ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >

1 0

Can't create new OSD
by Rodrigo Severo - Fábrica 12 Dec '19

12 Dec '19

Hi, Trying to create a new OSD following the instructions available at https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/ On step 3 I'm instructed to run "ceph-osd -i {osd-num} --mkfs --mkkey". Unfortunately it doesn't work: # ceph-osd -i 3 --mkfs --mkkey 2019-12-11 16:59:58.257 7fac4597fc00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-11 16:59:58.257 7fac4597fc00 -1 AuthRegistry(0x55ad976ea140) no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx 2019-12-11 16:59:58.261 7fac4597fc00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-11 16:59:58.261 7fac4597fc00 -1 AuthRegistry(0x7fffac4075e8) no keyring found at /var/lib/ceph/osd/ceph-3/keyring, disabling cephx failed to fetch mon config (--no-mon-config to skip) Shouldn't it create the keyring? Why is it complaining about not being able to find a keyring? Regards, Rodrigo

2 2

Ceph User Survey 2019
by Mike Perez 12 Dec '19

12 Dec '19

Hi Cephers, To better understand how our current users utilize Ceph, we conducted a public community survey. This information is a guide to the community of how we spend our contribution efforts for future development. The survey results will remain anonymous and aggregated in future Ceph Foundation publications to the community. I'm pleased to announce after much discussion on the Ceph dev mailing list [0] that the community has formed the Ceph Survey for 2019. The deadline for this survey due to it being out later than we'd like will be January 31st, 2020 at 11:59 PT. https://ceph.io/user-survey/ We have discussed in the future to use the Ceph telemetry module to collect the data to save time for our users. Please let me know of any mistakes that need to be corrected on the survey. Thanks! [0] - https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/WU374ZJP5N3NKY22X2… -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

4 3

Re: RESEND: Re: PG Balancer Upmap mode not working
by David Zafman 12 Dec '19

12 Dec '19

Philippe, I have a master branch version of the code to test. The nautilus backport https://github.com/ceph/ceph/pull/31956 should be the same. Using your OSDMap, the code in master branch and some additional changes to osdmaptool I was able to balance your cluster. The osdmaptool changes simulate the mgr active balancer behavior. It never took no more than 0.13991 seconds to calculate more upmaps per round. And that's on a virtual machine used for development. It took 35 rounds with 10 maximum upmaps per crush rule set of pools per round. With the default 1 minute sleeps inside the mgr it would take 35 minutes. Obviously, recovery/backfill has to finish before the cluster settles into the new configuration. It needed 397 additional upmaps and removed 8. Because all pools for a given crush rule are balanced together you can see that this is more balanced than Rich's configuration uising Luminous. This balancer code is subject to change before final release of the next Nautilus point release. Final layout: osd.0 pgs 146 osd.1 pgs 146 osd.2 pgs 146 osd.3 pgs 146 osd.4 pgs 146 osd.5 pgs 146 osd.6 pgs 146 osd.7 pgs 146 osd.8 pgs 146 osd.9 pgs 146 osd.10 pgs 146 osd.11 pgs 146 osd.12 pgs 74 osd.13 pgs 74 osd.14 pgs 73 osd.15 pgs 74 osd.16 pgs 74 osd.17 pgs 74 osd.18 pgs 73 osd.19 pgs 74 osd.20 pgs 73 osd.21 pgs 73 osd.22 pgs 74 osd.23 pgs 73 osd.24 pgs 73 osd.25 pgs 75 osd.26 pgs 74 osd.27 pgs 74 osd.28 pgs 73 osd.29 pgs 73 osd.30 pgs 73 osd.31 pgs 73 osd.32 pgs 74 osd.33 pgs 73 osd.34 pgs 73 osd.35 pgs 74 osd.36 pgs 74 osd.37 pgs 74 osd.38 pgs 74 osd.39 pgs 74 osd.40 pgs 73 osd.41 pgs 73 osd.42 pgs 73 osd.43 pgs 73 osd.44 pgs 74 osd.45 pgs 73 osd.46 pgs 73 osd.47 pgs 73 osd.48 pgs 73 osd.49 pgs 73 osd.50 pgs 73 osd.51 pgs 73 osd.52 pgs 75 osd.53 pgs 59 osd.54 pgs 74 osd.55 pgs 74 osd.56 pgs 74 osd.57 pgs 73 osd.58 pgs 74 osd.59 pgs 74 osd.60 pgs 74 osd.61 pgs 74 osd.62 pgs 73 osd.63 pgs 74 osd.64 pgs 73 osd.65 pgs 74 osd.66 pgs 74 osd.67 pgs 74 osd.68 pgs 73 osd.69 pgs 74 osd.70 pgs 73 osd.71 pgs 73 osd.72 pgs 73 osd.73 pgs 73 osd.74 pgs 73 osd.75 pgs 73 osd.76 pgs 73 osd.77 pgs 73 osd.78 pgs 73 osd.79 pgs 73 osd.80 pgs 73 osd.81 pgs 73 osd.82 pgs 73 osd.83 pgs 73 osd.84 pgs 73 osd.85 pgs 73 osd.86 pgs 73 osd.87 pgs 73 osd.88 pgs 73 osd.89 pgs 73 osd.90 pgs 73 osd.91 pgs 73 osd.92 pgs 73 osd.93 pgs 73 osd.94 pgs 73 osd.95 pgs 73 osd.96 pgs 73 osd.97 pgs 73 osd.98 pgs 73 osd.99 pgs 73 osd.100 pgs 146 osd.101 pgs 146 osd.102 pgs 146 osd.103 pgs 146 osd.104 pgs 146 osd.105 pgs 146 osd.106 pgs 146 osd.107 pgs 146 osd.108 pgs 146 osd.109 pgs 146 osd.110 pgs 146 osd.111 pgs 146 osd.112 pgs 73 osd.113 pgs 73 osd.114 pgs 73 osd.115 pgs 73 osd.116 pgs 73 osd.117 pgs 73 osd.118 pgs 73 osd.119 pgs 73 osd.120 pgs 73 osd.121 pgs 73 osd.122 pgs 73 osd.123 pgs 73 osd.124 pgs 73 osd.125 pgs 73 osd.126 pgs 73 osd.127 pgs 74 osd.128 pgs 73 osd.129 pgs 73 osd.130 pgs 73 osd.131 pgs 73 osd.132 pgs 73 osd.133 pgs 73 osd.134 pgs 73 osd.135 pgs 73 David On 12/10/19 9:59 PM, Philippe D'Anjou wrote: > Given I was told its an issue of too low PGs I am raising and testing > this, although my SSDs which have about 150 each also are not well > distributed. > I attached my OSDmap, I'd appreciate if you could run your test on it > like you did with the other guy, so I know if this will ever > distribute equally or not.. > > If you're busy I understand that too, then ignore this. > > Thanks in either case. Just have been dealing with this since months > now and it gets frustrating. > > best regards > > Am Dienstag, 10. Dezember 2019, 03:53:17 OEZ hat David Zafman > <dzafman(a)redhat.com> Folgendes geschrieben: > > > > Please file a tracker with the symptom and examples. Please attach your > OSDMap (ceph osd getmap > osdmap.bin). > > Note that https://github.com/ceph/ceph/pull/31956 > <https://github.com/ceph/ceph/pull/31956 >has the Nautilus > version of improved upmap code. It also changes osdmaptool to match the > mgr behavior, so that one can observe the behavior of the upmap balancer > offline. > > Thanks > > David > > On 12/8/19 11:04 AM, Philippe D'Anjou wrote: > > It's only getting worse after raising PGs now. > > > > Anything between: > > 96 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 97 KiB 13 GiB 4.2 > > TiB 53.62 0.76 54 up > > > > and > > > > 89 hdd 9.09470 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 88 KiB 21 GiB 1001 > > GiB 89.25 1.27 87 up > > > > How is that possible? I dont know how much more proof I need to > > present that there's a bug. > > > > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com <mailto:ceph-users@lists.ceph.com> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >

1 0

Size and capacity calculations questions
by Jochen Schulz 11 Dec '19

11 Dec '19

Hi! We have a ceph cluster with 42 OSD in production as a server providing mainly home-directories of users. Ceph is 14.2.4 nautilus. We have 3 pools. One images (for rbd images) a cephfs_metadata and a cephfs_data pool. Our raw data is about 5.6T. All pools have replica size 3 and there are only very little snapshots in the rbd images pool, the cephfspool doesnt use snapshots. How is it possible that the status tells us, that 21T/46T is used, because thats much more than 3 times the raw size. Also, to make that more confusing, there as at least half of the cluster free, and we get pg backfill_toofull after we added some OSDs lately. The ceph dashboard tells aus the pool ist 82 % full and has only 4.5 T free. The autoscale module seems to calculate the 20T times 3 for the space needed and thus has wrong numbers (see below). Status of the cluster is added below too. how can these size/capacity numbers be explained? and, would be there a recommendation to change something? Thank you in advance! best Jochen # ceph -s cluster: id: 2b16167f-3f33-4580-a0e9-7a71978f403d health: HEALTH_ERR Degraded data redundancy (low space): 1 pg backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 2 pools have too many placement groups services: mon: 4 daemons, quorum jade,assam,matcha,jasmine (age 2d) mgr: earl(active, since 24h), standbys: assam mds: cephfs:1 {0=assam=up:active} 1 up:standby osd: 42 osds: 42 up (since 106m), 42 in (since 115m); 30 remapped pgs data: pools: 3 pools, 2048 pgs objects: 29.80M objects, 5.6 TiB usage: 21 TiB used, 25 TiB / 46 TiB avail pgs: 1164396/89411013 objects misplaced (1.302%) 2018 active+clean 22 active+remapped+backfill_wait 7 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull io: client: 1.7 KiB/s rd, 516 KiB/s wr, 0 op/s rd, 28 op/s wr recovery: 9.2 MiB/s, 41 objects/s # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 354.2G 3.0 46100G 0.0231 1.0 1024 32 warn cephfs_metadata 13260M 3.0 595.7G 0.0652 1.0 512 8 warn cephfs_data 20802G 3.0 46100G 1.3537 1.0 512 warn

5 9

getfattr problem on ceph-fs
by Frank Schilder 11 Dec '19

11 Dec '19

I have a strange problem with ceph fs and extended attributes. I have two Centos machines where I mount cephfs in exactly the same way (I manually executed the exact same mount command on both machines). On one of the machines, getfattr returns this: [root@ceph-01 ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home getfattr: Removing leading '/' from absolute path names # file: mnt/cephfs/hpc/home ceph.dir.entries="49" ceph.dir.files="1" ceph.dir.rbytes="77816237666910" ceph.dir.rctime="1575978038.0976848840" ceph.dir.rentries="6673312" ceph.dir.rfiles="6271408" ceph.dir.rsubdirs="401904" ceph.dir.subdirs="48" and on the other I get nothing: [root@gnosis ~]# getfattr -d -m 'ceph.*' /mnt/cephfs/hpc/home No error message, just nothing. The only difference is, that ceph-01 was kickstarted with Centos7.6 while gnosis was kickstarted with Centos7.7. Otherwise, both machines are deployed identically. getfattr is the same version on both. Kernel versions are ceph-01:5.0.2-1.el7.elrepo.x86_64 and gnosis:5.4.2-1.el7.elrepo.x86_64. Does anyone have a pointer what to look for? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 4

Shouldn't Ceph's documentation be "per version"?
by Rodrigo Severo - Fábrica 10 Dec '19

10 Dec '19

Hi, Shouldn't Ceph's documentation be presented "per version"? I believe there might be documentation for Ceph per version but I can't see in Ceph documentation site how to easily see each version's docs. Regards, Rodrigo Severo

2 1

Cephalocon 2020
by Sage Weil 10 Dec '19

10 Dec '19

Hi everyone, The next Cephalocon is coming up on March 3-5 in Seoul! The CFP is open until Friday (get your talks in!). We expect to have the program ready for the first week of January. Registration (early bird) will be available soon. We're also looking for sponsors for the conference. The prospectus is available here: https://ceph.io/wp-content/uploads/2019/12/sponsor-Cephalocon20-112719.pdf Thanks! sage

1 1

[object gateway] setting storage class does not move object to correct backing pool?
by Gerdriaan Mulder 10 Dec '19

10 Dec '19

Hi, If I change the storage class of an object via s3cmd, the object's storage class is reported as being changed. However, when inspecting where the objects are placed (via `rados -p <pool> ls`, see further on), the object seems to be retained in the original pool. The idea behind this test setup is to simulate two storage locations, one based on SSDs or similar flash storage, the other on slow HDDs. We want to be able to alter the storage location of objects on the fly, typically only from fast to slow storage. The object should then only reside on slow storage. The setup is as follows on Nautilus (Ubuntu 16.04, see <https://gist.github.com/mrngm/bba6ffdc545bfa52ebf79d6d2c002a6d> for the full dump): <<<<<<<< root@node1:~# ceph -s health: HEALTH_OK mon: 3 daemons, quorum node1,node3,node5 (age 12d) mgr: node2(active, since 6d), standbys: node4 osd: 4 osds: 4 up (since 12d), 4 in (since 12d) rgw: 1 daemon active (node1) pools: 7 pools, 296 pgs objects: 229 objects, 192 KiB usage: 3.2 GiB used, 6.8 GiB / 10 GiB avail pgs: 296 active+clean root@node1:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.00970 root default -16 0.00970 datacenter nijmegen -3 0.00388 host node2 0 hdd 0.00388 osd.0 up 1.00000 1.00000 -5 0.00388 host node3 1 hdd 0.00388 osd.1 up 1.00000 1.00000 -7 0.00098 host node4 2 ssd 0.00098 osd.2 up 1.00000 1.00000 -9 0.00098 host node5 3 ssd 0.00098 osd.3 up 1.00000 1.00000 root@node1:~# ceph osd pool ls detail pool 1 'tier1-ssd' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 [snip] application rgw pool 2 'tier2-hdd' replicated size 1 min_size 1 crush_rule 2 object_hash rjenkins pg_num 128 pgp_num 128 [snip] application rgw pool 3 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 [snip] application rgw pool 4 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 [snip] application rgw pool 5 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 [snip] application rgw pool 6 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 [snip] application rgw pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 [snip] application rgw root@node1:~# ceph osd pool application get # compacted tier1-ssd => rgw {} tier2-hdd => rgw {} .rgw.root => rgw {} default.rgw.control => rgw {} default.rgw.meta => rgw {} default.rgw.log => rgw {} default.rgw.buckets.index => rgw {} root@node1:~# radosgw-admin zonegroup placement list [ { "key": "default-placement", "val": { "name": "default-placement", "tags": [], "storage_classes": [ "SPINNING_RUST", "STANDARD" ] } } ] root@node1:~# radosgw-admin zone placement list [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "SPINNING_RUST": { "data_pool": "tier2-hdd" }, "STANDARD": { "data_pool": "tier1-ssd" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } } ] ======== I can also post the relevant s3cmd commands for putting objects and setting the storage class, but perhaps this is already enough information. Please let me know. <<<<<<<< root@node1:~# rados -p tier1-ssd ls ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_darthvader.png ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_2019-10-15-090436_1254x522_scrubbed.png ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1_kanariepiet.jpg root@node1:~# rados -p tier2-hdd ls ce2fc9ee-edc8-4dc7-a3fe-b1458c67168b.5805.1__shadow_.FEruUOZaVJXJcOG-e2tO1xcInNzoEvN_0 $ s3cmd info s3://bucket/kanariepiet.jpg [snip] Last mod: Tue, 10 Dec 2019 08:09:58 GMT Storage: STANDARD [snip] $ s3cmd info s3://bucket/darthvader.png [snip] Last mod: Wed, 04 Dec 2019 10:35:14 GMT Storage: SPINNING_RUST [snip] $ s3cmd info s3://bucket/2019-10-15-090436_1254x522_scrubbed.png [snip] Last mod: Tue, 10 Dec 2019 10:33:24 GMT Storage: STANDARD [snip] ========== Any thoughts on what might occur here? Best regards, Gerdriaan Mulder

3 4

nautilus radosgw fails with pre jewel buckets - index objects not at right place
by Ingo Reimann 10 Dec '19

10 Dec '19

Hi, 2 years after my issue [ https://tracker.ceph.com/issues/22928 | https://tracker.ceph.com/issues/22928 ] the next one fires back. The Problem: Old Buckets have their index and data in rgw.buckets: root@cephrgw01:~# radosgw-admin metadata get bucket:testtesttesty { "key": "bucket:testtesttesty", "ver": { "tag": "_E_OHNhD28Zu1DeuvyGq8Q8b", "ver": 1 }, "mtime": "2013-11-11 09:25:56.000000Z", "data": { "bucket": { "name": "testtesttesty", "marker": "default.2542971.19", "bucket_id": "default.2542971.19", "tenant": "", "explicit_placement": { "data_pool": "rgw.buckets", "data_extra_pool": "", "index_pool": "rgw.buckets" } }, "owner": "123", "creation_time": "2013-11-11 09:25:56.000000Z", "linked": "true", "has_bucket_info": "false" } } After upgrade from luminous to nautilus i get 400(InvalidArgument) and "NOTICE: invalid dest placement" in the radosgw-log on access to the buckets My zone defines: root@cephrgw01:~# radosgw-admin zone get { "id": "default", "name": "default", "domain_root": ".rgw", "control_pool": ".rgw.control", "gc_pool": ".rgw.gc", "lc_pool": ".log:lc", "log_pool": ".log", "intent_log_pool": ".intent-log", "usage_log_pool": ".usage", "reshard_pool": ".log:reshard", "user_keys_pool": ".users", "user_email_pool": ".users.email", "user_swift_pool": ".users.swift", "user_uid_pool": ".users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw.buckets" } }, "data_extra_pool": "rgw.buckets.non-ec", "index_type": 0 } } ], "metadata_heap": ".rgw.meta", "realm_id": "*********************c" } Now i am a little bit lost. I added a new placement to my zone and zonegroup radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id pre-jewel radosgw-admin zone placement add --rgw-zonegroup default --placement-id pre-jewel --data-pool rgw.buckets --index-pool rgw.buckets data-extra-pool "" radosgw-admin period update --commit root@cephrgw01:~# radosgw-admin zone get { "id": "default", "name": "default", "domain_root": ".rgw", "control_pool": ".rgw.control", "gc_pool": ".rgw.gc", "lc_pool": ".log:lc", "log_pool": ".log", "intent_log_pool": ".intent-log", "usage_log_pool": ".usage", "reshard_pool": ".log:reshard", "user_keys_pool": ".users", "user_email_pool": ".users.email", "user_swift_pool": ".users.swift", "user_uid_pool": ".users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw.buckets" } }, "data_extra_pool": "rgw.buckets.non-ec", "index_type": 0 } }, { "key": "pre-jewel", "val": { "index_pool": "rgw.buckets", "storage_classes": { "STANDARD": { "data_pool": "rgw.buckets" } }, "data_extra_pool": "", "index_type": 0 } } ], "metadata_heap": ".rgw.meta", "realm_id": "****************c" } Nevertheless, only the luminous gateways may list my old buckets. As far as I can see, I may only change the placement_rule for new buckets. Is there any chance to make radosgw find the old indices and complete the upgrade to nautilus? Many thanks, Ingo -- Ingo Reimann [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

2 5

Re: osdmaps not trimmed until ceph-mon's
by Romit Misra 10 Dec '19

10 Dec '19

Hi Bryan and Dan, Had some similar observations, and wanted some data points from you as well if possible. 1. When you say down osd , is it down and in or down and out. 2. I see the osd accumulating map ranges, when continuous recovery is going on, due to osd flaps( that is some portion of the total of pgs in cluster are not in active+clean) state. 3.Another observation is that, although it may be one of the pools that see a osd churn, the map ranges seem to increase across all the OSd across all the pools. 4.I use the asok socket to dump the status, on osd Daemon which shows the map ranges. 5.Even I tried restarting Mon, one after another, and that seemed to reduce the space, but in a sporadic fashion. 6. Once there was some breathing room, in terms of OSd not flapping, and all PG being in active + clean state, the size trimmed down to almost optimal spaces. 7. I found some pointers at :- https://docs.ceph.com/docs/master/dev/mon-osdmap-prune/ 8. Th behaviour I am specifying us in 12.X Luminous release. 9. Would you care to share what commands you use to dump maps usage info from space perspective. Thanks Romit On Mon, 9 Dec 2019, 22:56 , <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. RGW listing millions of objects takes too much time (Arash Shams) > 2. Re: RGW listing millions of objects takes too much time > (Robert LeBlanc) > 3. ceph mgr daemon multiple ip addresses (Frank R) > 4. Re: osdmaps not trimmed until ceph-mon's restarted (if cluster has a > down osd) > (Bryan Stillwell) > > > ---------------------------------------------------------------------- > > Date: Mon, 9 Dec 2019 15:46:04 +0000 > From: Arash Shams <ara4sh(a)hotmail.com> > Subject: [ceph-users] RGW listing millions of objects takes too much > time > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <CWXP265MB128679DE9E6B2A6CA289B16492580(a)CWXP265MB1286.GBR > P265.PROD.OUTLOOK.COM> > Content-Type: multipart/alternative; boundary="_000_CWXP265MB128679 > DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_" > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > Dear All, > > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, > Im using boto3 with continuation Marker but it takes almost 9 hours > > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, > > > Thanks > > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/html; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > <html> > <head> > <meta http-equiv=3D"Content-Type" content=3D"text/html; > charset=3Diso-8859-= > 1"> > <style type=3D"text/css" style=3D"display:none;"> P > {margin-top:0;margin-bo= > ttom:0;} </style> > </head> > <body dir=3D"ltr"> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Dear All, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Thanks </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > </body> > </html> > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 08:23:55 -0800 > From: Robert LeBlanc <robert(a)leblancnet.us> > Subject: [ceph-users] Re: RGW listing millions of objects takes too > much time > To: Arash Shams <ara4sh(a)hotmail.com> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: > <CAANLjFoZu8LE0eH2vB9QArAOZJf4Ofm8V= > jAW-+ZPOFS-f9Oag(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000000e664e059947d3b2" > > --0000000000000e664e059947d3b2 > Content-Type: text/plain; charset="UTF-8" > > On Mon, Dec 9, 2019 at 7:47 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > > Dear All, > > > > I have almost 30 million objects and I want to list them and index them > > somewhere else, > > Im using boto3 with continuation Marker but it takes almost 9 hours > > > > can I run it in multiple threads to make it faster? what solution do you > > suggest to speedup this process, > > > > > > Thanks > > > > I've thought about indexing objects elsewhere as well. One thought I had > was hooking into the HTTP flow where a PUT or DEL would update the objects > in some kind of database (async of course). We could also gather stats with > GET and POST. Initially, my thoughts were to hook into haproxy since we > already use it, but possibly RGW if that is an option. That way it would > always be up to date and not have to do big scans on the buckets (our > buckets would not perform well with this). I haven't actually gotten to the > implementation phase of this idea. > > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > --0000000000000e664e059947d3b2 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div dir=3D"ltr">On Mon, Dec 9, 2019 at 7:47 AM Arash > Sham= > s <<a href=3D"mailto:ara4sh@hotmail.com">ara4sh(a)hotmail.com</a>> > wrot= > e:<br></div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" > st= > yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid > rgb(204,204,204);padd= > ing-left:1ex"> > > > > > <div dir=3D"ltr"> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Dear All,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Thanks=C2=A0</div></div></blockquote><div > class=3D"gmail_quote"><br></div><= > div class=3D"gmail_quote">I've thought about indexing objects > elsewhere= > as well. One thought I had was hooking into the HTTP flow where a PUT or > D= > EL would update the objects in some kind of database (async of course). We > = > could also gather stats with GET and POST. Initially, my thoughts were to > h= > ook into haproxy since we already use it, but possibly RGW if that is an > op= > tion. That way it would always be up to date and not have to do big scans > o= > n the buckets (our buckets would not perform well with this). I > haven't= > actually gotten to the implementation phase of this idea.</div><br clear= > =3D"all"><div><div dir=3D"ltr" > class=3D"gmail_signature">----------------<b= > r>Robert LeBlanc<br></div></div><div>PGP Fingerprint 79A2 9CA4 6CC4 45DD > A9= > 04 =C2=A0C70E E654 3BB2 FA62 B9F1=C2=A0</div></div></div> > > --0000000000000e664e059947d3b2-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 11:54:10 -0500 > From: Frank R <frankaritchie(a)gmail.com> > Subject: [ceph-users] ceph mgr daemon multiple ip addresses > To: ceph-users <ceph-users(a)ceph.com> > Message-ID: > <CAMuVLDOhax+8RwPEK_52UfzmOc8dv5= > YBH7Etsm5VmiO_WFEPg(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000004ef36d0599483fd9" > > --0000000000004ef36d0599483fd9 > Content-Type: text/plain; charset="UTF-8" > > Hi all, > > Does anyone know what possible issues can arise if the ceph mgr daemon is > running on a mon node that has 2 ips in the public net range (1 is a > loopback address). > > As I understand the it. mgr will bind to all ips > > FYI - I am not sure why the loopback is there, I am trying to find out. > > thx > Frank > > > > > mlovell - ceph anycast > > --0000000000004ef36d0599483fd9 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div>Hi all,</div><div><br>Does anyone know what possible > = > issues can arise if the ceph mgr daemon is running on a mon node that has > 2= > ips in the public net range (1 is a loopback > address).<br><br></div><div>A= > s I understand the it. mgr will bind to all ips<br><br></div><div>FYI - I > a= > m not sure why the loopback is there, I am trying to find > out.</div><div><b= > r></div><div>thx</div><div>Frank<br><br><br><br><br>mlovell - ceph > anycast<= > br></div></div> > > --0000000000004ef36d0599483fd9-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 17:24:27 +0000 > From: Bryan Stillwell <bstillwell(a)godaddy.com> > Subject: [ceph-users] Re: osdmaps not trimmed until ceph-mon's > restarted (if cluster has a down osd) > To: Dan van der Ster <dan(a)vanderster.com> > Cc: Joao Eduardo Luis <joao(a)suse.de>, "dev(a)ceph.io" <dev(a)ceph.io>, > ceph-users <ceph-users(a)ceph.io> > Message-ID: <9B145B17-6665-4254-8D6A-04A9B37389C3(a)godaddy.com> > Content-Type: text/plain; charset="us-ascii" > > On Nov 18, 2019, at 8:12 AM, Dan van der Ster <dan(a)vanderster.com> wrote: > > > > On Fri, Nov 15, 2019 at 4:45 PM Joao Eduardo Luis <joao(a)suse.de> wrote: > >> > >> On 19/11/14 11:04AM, Gregory Farnum wrote: > >>> On Thu, Nov 14, 2019 at 8:14 AM Dan van der Ster <dan(a)vanderster.com> > wrote: > >>>> > >>>> Hi Joao, > >>>> > >>>> I might have found the reason why several of our clusters (and maybe > >>>> Bryan's too) are getting stuck not trimming osdmaps. > >>>> It seems that when an osd fails, the min_last_epoch_clean gets stuck > >>>> forever (even long after HEALTH_OK), until the ceph-mons are > >>>> restarted. > >>>> > >>>> I've updated the ticket: https://tracker.ceph.com/issues/41154 > >>> > >>> Wrong ticket, I think you meant > https://tracker.ceph.com/issues/37875#note-7 > >> > >> I've seen this behavior a long, long time ago, but stopped being able to > >> reproduce it consistently enough to ensure the patch was working > properly. > >> > >> I think I have a patch here: > >> > >> https://github.com/ceph/ceph/pull/19076/commits > >> > >> If you are feeling adventurous, and want to give it a try, let me know. > I'll > >> be happy to forward port it to whatever you are running. > > > > Thanks Joao, this patch is what I had in mind. > > > > I'm trying to evaluate how adventurous this would be -- Is there any > > risk that if a huge number of osds are down all at once (but > > transiently), it would trigger the mon to trim too many maps? > > I would expect that the remaining up OSDs will have a safe, low, > osd_epoch ? > > > > And anyway I guess that your proposed get_min_last_epoch_clean patch > > is equivalent to what we have today if we restart the ceph-mon leader > > while an osd is down. > > Joao, > > I ran into this again today and found over 100,000 osdmaps on all 1,000 > OSDs (~50 TiB of disk space used just by osdmaps). There were down OSDs > (pretty regular occurrence with ~1,000 OSDs) so that matches up with what > Dan found. Then when I restarted all the mon nodes twice the osdmaps > started cleaning up. > > I believe the steps to reproduce would look like this: > > 1. Start with a cluster with at least 1 down osd > 2. Expand the cluster (the bigger the expansion, the more osdmaps that > pile up) > 3. Notice that after the expansion completes and the cluster is healthy > that the old osdmaps aren't cleaned up > > I would be willing to test the fix on our test cluster after 14.2.5 comes > out. Could you make a build based on that release? > > Thanks, > Bryan > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 83, Issue 35 > ****************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

Re: : RGW listing millions of objects takes too much time
by Romit Misra 10 Dec '19

10 Dec '19

Hi Robert and Arash, A couple of pointers and asks that might help. 1. Can you point to the code you are using for listing the buckets. 2. Which release is the cluster running on..? 3. What is the number of shards that have been configured in the bucket index for the said mentioned bucket...? 4.Have you tried timing the listing process, as to where does the slow down began, in other words after 1 million entries, 2 million entri s, or the listing is slow at per key level. 5. You could profile the timing for every 10000 entries or 100000 6.While using a additional meta, like redis in memory meta store could be a solution, but moving many moving parts altogether becomes a headache. 7. Let's look at the tunebales first, to see where the bottleneck is. 8.Another good practice , of possible in your deployment would be, if you can restrict objects per bucket, to say .5 million or 1 million, and have a client side hash/placement algorithm for writing into the buckets. 9. If these 30 million entries are static, iff , then prefetching the markers , and running are multithreaded flow makes sense. But again if these are static why would you not store them in a blob altogether, so contraindicating my own point. 10.Also do look ceph lists and mailers for ordered Vs unordered listing(remember seeing some query around it) Thanks Romit Misra On Mon, 9 Dec 2019, 22:56 , <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. RGW listing millions of objects takes too much time (Arash Shams) > 2. Re: RGW listing millions of objects takes too much time > (Robert LeBlanc) > 3. ceph mgr daemon multiple ip addresses (Frank R) > 4. Re: osdmaps not trimmed until ceph-mon's restarted (if cluster has a > down osd) > (Bryan Stillwell) > > > ---------------------------------------------------------------------- > > Date: Mon, 9 Dec 2019 15:46:04 +0000 > From: Arash Shams <ara4sh(a)hotmail.com> > Subject: [ceph-users] RGW listing millions of objects takes too much > time > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <CWXP265MB128679DE9E6B2A6CA289B16492580(a)CWXP265MB1286.GBR > P265.PROD.OUTLOOK.COM> > Content-Type: multipart/alternative; boundary="_000_CWXP265MB128679 > DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_" > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > Dear All, > > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, > Im using boto3 with continuation Marker but it takes almost 9 hours > > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, > > > Thanks > > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_ > Content-Type: text/html; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > <html> > <head> > <meta http-equiv=3D"Content-Type" content=3D"text/html; > charset=3Diso-8859-= > 1"> > <style type=3D"text/css" style=3D"display:none;"> P > {margin-top:0;margin-bo= > ttom:0;} </style> > </head> > <body dir=3D"ltr"> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Dear All, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process, </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Thanks </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > <br> > </div> > </body> > </html> > > --_000_CWXP265MB128679DE9E6B2A6CA289B16492580CWXP265MB1286GBRP_-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 08:23:55 -0800 > From: Robert LeBlanc <robert(a)leblancnet.us> > Subject: [ceph-users] Re: RGW listing millions of objects takes too > much time > To: Arash Shams <ara4sh(a)hotmail.com> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: > <CAANLjFoZu8LE0eH2vB9QArAOZJf4Ofm8V= > jAW-+ZPOFS-f9Oag(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000000e664e059947d3b2" > > --0000000000000e664e059947d3b2 > Content-Type: text/plain; charset="UTF-8" > > On Mon, Dec 9, 2019 at 7:47 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > > Dear All, > > > > I have almost 30 million objects and I want to list them and index them > > somewhere else, > > Im using boto3 with continuation Marker but it takes almost 9 hours > > > > can I run it in multiple threads to make it faster? what solution do you > > suggest to speedup this process, > > > > > > Thanks > > > > I've thought about indexing objects elsewhere as well. One thought I had > was hooking into the HTTP flow where a PUT or DEL would update the objects > in some kind of database (async of course). We could also gather stats with > GET and POST. Initially, my thoughts were to hook into haproxy since we > already use it, but possibly RGW if that is an option. That way it would > always be up to date and not have to do big scans on the buckets (our > buckets would not perform well with this). I haven't actually gotten to the > implementation phase of this idea. > > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > --0000000000000e664e059947d3b2 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div dir=3D"ltr">On Mon, Dec 9, 2019 at 7:47 AM Arash > Sham= > s <<a href=3D"mailto:ara4sh@hotmail.com">ara4sh(a)hotmail.com</a>> > wrot= > e:<br></div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" > st= > yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid > rgb(204,204,204);padd= > ing-left:1ex"> > > > > > <div dir=3D"ltr"> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Dear All,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > I have almost 30 million objects and I want to list them and index them > som= > ewhere else,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Im using boto3 with continuation Marker but it takes almost 9 hours</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > can I run it in multiple threads to make it faster? what solution do you > su= > ggest to speedup this process,=C2=A0</div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > <br> > </div> > <div > style=3D"font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color= > :rgb(0,0,0)"> > Thanks=C2=A0</div></div></blockquote><div > class=3D"gmail_quote"><br></div><= > div class=3D"gmail_quote">I've thought about indexing objects > elsewhere= > as well. One thought I had was hooking into the HTTP flow where a PUT or > D= > EL would update the objects in some kind of database (async of course). We > = > could also gather stats with GET and POST. Initially, my thoughts were to > h= > ook into haproxy since we already use it, but possibly RGW if that is an > op= > tion. That way it would always be up to date and not have to do big scans > o= > n the buckets (our buckets would not perform well with this). I > haven't= > actually gotten to the implementation phase of this idea.</div><br clear= > =3D"all"><div><div dir=3D"ltr" > class=3D"gmail_signature">----------------<b= > r>Robert LeBlanc<br></div></div><div>PGP Fingerprint 79A2 9CA4 6CC4 45DD > A9= > 04 =C2=A0C70E E654 3BB2 FA62 B9F1=C2=A0</div></div></div> > > --0000000000000e664e059947d3b2-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 11:54:10 -0500 > From: Frank R <frankaritchie(a)gmail.com> > Subject: [ceph-users] ceph mgr daemon multiple ip addresses > To: ceph-users <ceph-users(a)ceph.com> > Message-ID: > <CAMuVLDOhax+8RwPEK_52UfzmOc8dv5= > YBH7Etsm5VmiO_WFEPg(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000004ef36d0599483fd9" > > --0000000000004ef36d0599483fd9 > Content-Type: text/plain; charset="UTF-8" > > Hi all, > > Does anyone know what possible issues can arise if the ceph mgr daemon is > running on a mon node that has 2 ips in the public net range (1 is a > loopback address). > > As I understand the it. mgr will bind to all ips > > FYI - I am not sure why the loopback is there, I am trying to find out. > > thx > Frank > > > > > mlovell - ceph anycast > > --0000000000004ef36d0599483fd9 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr"><div>Hi all,</div><div><br>Does anyone know what possible > = > issues can arise if the ceph mgr daemon is running on a mon node that has > 2= > ips in the public net range (1 is a loopback > address).<br><br></div><div>A= > s I understand the it. mgr will bind to all ips<br><br></div><div>FYI - I > a= > m not sure why the loopback is there, I am trying to find > out.</div><div><b= > r></div><div>thx</div><div>Frank<br><br><br><br><br>mlovell - ceph > anycast<= > br></div></div> > > --0000000000004ef36d0599483fd9-- > > ------------------------------ > > Date: Mon, 9 Dec 2019 17:24:27 +0000 > From: Bryan Stillwell <bstillwell(a)godaddy.com> > Subject: [ceph-users] Re: osdmaps not trimmed until ceph-mon's > restarted (if cluster has a down osd) > To: Dan van der Ster <dan(a)vanderster.com> > Cc: Joao Eduardo Luis <joao(a)suse.de>, "dev(a)ceph.io" <dev(a)ceph.io>, > ceph-users <ceph-users(a)ceph.io> > Message-ID: <9B145B17-6665-4254-8D6A-04A9B37389C3(a)godaddy.com> > Content-Type: text/plain; charset="us-ascii" > > On Nov 18, 2019, at 8:12 AM, Dan van der Ster <dan(a)vanderster.com> wrote: > > > > On Fri, Nov 15, 2019 at 4:45 PM Joao Eduardo Luis <joao(a)suse.de> wrote: > >> > >> On 19/11/14 11:04AM, Gregory Farnum wrote: > >>> On Thu, Nov 14, 2019 at 8:14 AM Dan van der Ster <dan(a)vanderster.com> > wrote: > >>>> > >>>> Hi Joao, > >>>> > >>>> I might have found the reason why several of our clusters (and maybe > >>>> Bryan's too) are getting stuck not trimming osdmaps. > >>>> It seems that when an osd fails, the min_last_epoch_clean gets stuck > >>>> forever (even long after HEALTH_OK), until the ceph-mons are > >>>> restarted. > >>>> > >>>> I've updated the ticket: https://tracker.ceph.com/issues/41154 > >>> > >>> Wrong ticket, I think you meant > https://tracker.ceph.com/issues/37875#note-7 > >> > >> I've seen this behavior a long, long time ago, but stopped being able to > >> reproduce it consistently enough to ensure the patch was working > properly. > >> > >> I think I have a patch here: > >> > >> https://github.com/ceph/ceph/pull/19076/commits > >> > >> If you are feeling adventurous, and want to give it a try, let me know. > I'll > >> be happy to forward port it to whatever you are running. > > > > Thanks Joao, this patch is what I had in mind. > > > > I'm trying to evaluate how adventurous this would be -- Is there any > > risk that if a huge number of osds are down all at once (but > > transiently), it would trigger the mon to trim too many maps? > > I would expect that the remaining up OSDs will have a safe, low, > osd_epoch ? > > > > And anyway I guess that your proposed get_min_last_epoch_clean patch > > is equivalent to what we have today if we restart the ceph-mon leader > > while an osd is down. > > Joao, > > I ran into this again today and found over 100,000 osdmaps on all 1,000 > OSDs (~50 TiB of disk space used just by osdmaps). There were down OSDs > (pretty regular occurrence with ~1,000 OSDs) so that matches up with what > Dan found. Then when I restarted all the mon nodes twice the osdmaps > started cleaning up. > > I believe the steps to reproduce would look like this: > > 1. Start with a cluster with at least 1 down osd > 2. Expand the cluster (the bigger the expansion, the more osdmaps that > pile up) > 3. Notice that after the expansion completes and the cluster is healthy > that the old osdmaps aren't cleaned up > > I would be willing to test the fix on our test cluster after 14.2.5 comes > out. Could you make a build based on that release? > > Thanks, > Bryan > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 83, Issue 35 > ****************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

High swap usage on one replication node
by Götz Reinicke 09 Dec '19

09 Dec '19

Hi, our Ceph 14.2.3 cluster so far runs smooth with replicated and EC pools, but since a couple of days one of the dedicated replication nodes consumes up to 99% swap and stays at that level. The other two replicated nodes use +- 50 - 60% of swap. All the 24 NVMe OSDs per node are BlueStore with default settings, 128GB RAM. The vm.swappiness is set to 10. Do you have any suggestions how to handle/reduce the swap usage? Thanks for feedback and regards . Götz

5 5

ceph mgr daemon multiple ip addresses
by Frank R 09 Dec '19

09 Dec '19

Hi all, Does anyone know what possible issues can arise if the ceph mgr daemon is running on a mon node that has 2 ips in the public net range (1 is a loopback address). As I understand the it. mgr will bind to all ips FYI - I am not sure why the loopback is there, I am trying to find out. thx Frank mlovell - ceph anycast

2 1

osdmaps not trimmed until ceph-mon's restarted (if cluster has a down osd)
by Dan van der Ster 09 Dec '19

09 Dec '19

Hi Joao, I might have found the reason why several of our clusters (and maybe Bryan's too) are getting stuck not trimming osdmaps. It seems that when an osd fails, the min_last_epoch_clean gets stuck forever (even long after HEALTH_OK), until the ceph-mons are restarted. I've updated the ticket: https://tracker.ceph.com/issues/41154 Cheers, Dan

4 4

Re: RGW listing millions of objects takes too much time
by Robert LeBlanc 09 Dec '19

09 Dec '19

On Mon, Dec 9, 2019 at 7:47 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > Dear All, > > I have almost 30 million objects and I want to list them and index them > somewhere else, > Im using boto3 with continuation Marker but it takes almost 9 hours > > can I run it in multiple threads to make it faster? what solution do you > suggest to speedup this process, > > > Thanks > I've thought about indexing objects elsewhere as well. One thought I had was hooking into the HTTP flow where a PUT or DEL would update the objects in some kind of database (async of course). We could also gather stats with GET and POST. Initially, my thoughts were to hook into haproxy since we already use it, but possibly RGW if that is an option. That way it would always be up to date and not have to do big scans on the buckets (our buckets would not perform well with this). I haven't actually gotten to the implementation phase of this idea. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

1 0

RGW listing millions of objects takes too much time
by Arash Shams 09 Dec '19

09 Dec '19

Dear All, I have almost 30 million objects and I want to list them and index them somewhere else, Im using boto3 with continuation Marker but it takes almost 9 hours can I run it in multiple threads to make it faster? what solution do you suggest to speedup this process, Thanks

1 0

OSD state<Start>: transitioning to Stray
by Thomas Schneider 09 Dec '19

09 Dec '19

Hi, I had a failure on 2 of 7 OSD nodes. This caused a server reboot and unfortunately the cluster network failed to come up. This resulted in many OSD down situation. I decided to stop all services (OSD, MGR, MON) and to start them sequentially. Now I have multiple OSD marked as down although the service is running. None of these down OSDS is connected to the 2 nodes with failure. In the OSD logs I can see multiple entries like this: 2019-12-09 11:13:10.378 7f9a372fb700 1 osd.374 pg_epoch: 493189 pg[11.1992( v 457986'92619 (303558'88266,457986'92619] local-lis/les=466724/466725 n=4107 ec=8346/8346 lis/c 466724/466724 les/c/f 466725/466725/176266 468956/493184/468423) [203,412] r=-1 lpr=493184 pi=[466724,493184)/1 crt=457986'92619 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray I tried to restart the impacted OSD w/o success, means the relevant OSD is still marked as down. Is there a procedure to overcome this issue, means getting all OSD up? THX

2 4

Multi-site RadosGW with multiple placement targets
by Tobias Urdin 09 Dec '19

09 Dec '19

Hello, I'm trying to wrap my head around how having a multi-site (two zones in one zonegroup) with multiple placement targets but only wanting to replicate some placement targets would work. Can you setup a zonegroup with two zones and it would only replicate the placement targets that the other side has? For example if we have three placement targets in the zonegroup, one zone got all three placement targets and the other zone only has two of them, would it only replicate the two? Best regards

3 3

ceph public network definition
by Frank R 07 Dec '19

07 Dec '19

Is the public_network definition in ceph.conf just used to determine which interface or IP to use for the public network or does it need to need to encompass the public ip addresses of all cluster nodes? Specifically, can the public_network be defined differently for different OSD nodes as long as they have layer 3 connectivity? thx Frank

1 0

rbd_open_by_id crash when connection timeout
by yangjun＠cmss.chinamobile.com 07 Dec '19

07 Dec '19

Hi Jason, dongsheng I found a problem using rbd_open_by_id when connection timeout(errno = 110, ceph version 12.2.8, there is no change about rbd_open_by_id in master branch). int r = ictx->state->open(false); if (r < 0) { // r = -110 delete ictx; // crash，the stack is shown below： } else { *image = (rbd_image_t)ictx; } Stack is ： (gdb) bt Program terminated with signal 11, Segmentation fault. #0 Mutex::Lock (this=this@entry=0x8, no_lockdep=no_lockdep@entry=false) at /var/lib/jenkins/workspace/ceph_L/ceph/src/common/Mutex.cc:92 #1 0x0000ffff765eb814 in Locker (m=..., this=<synthetic pointer>) at /var/lib/jenkins/workspace/ceph_L/ceph/src/common/Mutex.h:115 #2 PerfCountersCollection::remove (this=0x0, l=0xffff85bf0588 <main_arena+88>) at /var/lib/jenkins/workspace/ceph_L/ceph/src/common/perf_counters.cc:62 #3 0x0000ffff757f26c4 in librbd::ImageCtx::perf_stop (this=0x2065d2f0) at /var/lib/jenkins/workspace/ceph_L/ceph/src/librbd/ImageCtx.cc:426 #4 0x0000ffff757f6e48 in librbd::ImageCtx::~ImageCtx (this=0x2065d2f0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/ceph_L/ceph/src/librbd/ImageCtx.cc:239 #5 0x0000ffff757e8e70 in rbd_open_by_id (p=p@entry=0x2065e780, id=id@entry=0xffff85928bb4 "20376aa706c0", image=image@entry=0xffff75b9b0b8, snap_name=snap_name@entry=0x0) at /var/lib/jenkins/workspace/ceph_L/ceph/src/librbd/librbd.cc:2692 #6 0x0000ffff75aed524 in __pyx_pf_3rbd_5Image___init__ ( __pyx_v_read_only=0xffff85f02bb8 <_Py_ZeroStruct>, __pyx_v_snapshot=0xffff85f15938 <_Py_NoneStruct>, __pyx_v_by_id=0xffff85f02ba0 <_Py_TrueStruct>, __pyx_v_name=0xffff85928b90, __pyx_v_ioctx=0xffff7f0283d0, __pyx_v_self=0xffff75b9b0a8) at /var/lib/jenkins/workspace/ceph_L/ceph/build/src/pybind/rbd/pyrex/rbd.c:12662 #7 __pyx_pw_3rbd_5Image_1__init__ (__pyx_v_self=0xffff75b9b0a8, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at /var/lib/jenkins/workspace/ceph_L/ceph/build/src/pybind/rbd/pyrex/rbd.c:12378 #8 0x0000ffff85e02924 in type_call () from /lib64/libpython2.7.so.1.0 #9 0x0000ffff85dac254 in PyObject_Call () from /lib64/libpython2.7.so.1.0 Meanwhile, I have sone questions about rbd_open, if state->open failed（return value r < 0）, Is there a risk of memory leaks about ictx? extern "C" int rbd_open(rados_ioctx_t p, const char *name, rbd_image_t *image, const char *snap_name) { librados::IoCtx io_ctx; librados::IoCtx::from_rados_ioctx_t(p, io_ctx); TracepointProvider::initialize<tracepoint_traits>(get_cct(io_ctx)); librbd::ImageCtx *ictx = new librbd::ImageCtx(name, "", snap_name, io_ctx, false); tracepoint(librbd, open_image_enter, ictx, ictx->name.c_str(), ictx->id.c_str(), ictx->snap_name.c_str(), ictx->read_only); int r = ictx->state->open(false); if (r >= 0) { // if r < 0, Is there a risk of memory leaks? *image = (rbd_image_t)ictx; } tracepoint(librbd, open_image_exit, r); return r; } Can you give me some advice? many thanks. yangjun(a)cmss.chinamobile.com

3 6

Upgrade from Jewel to Nautilus
by 徐蕴 07 Dec '19

07 Dec '19

Hello, We are planning to upgrade our cluster from Jewel to Nautilus. From my understanding, leveldb of monitor and filestore of OSDs will not be converted to rocketdb and bluestore automatically. So do you suggest to convert them manually after upgrading software? Is there any document or guidance available? Br, Xu Yun

2 2

Starting service rbd-target-api fails
by Thomas Schneider 06 Dec '19

06 Dec '19

Hi, I want to setup Ceph iSCSI Gateway and I follow this <https://docs.ceph.com/docs/master/rbd/iscsi-overview/> documentation. In step "Setup" of process "Configuring the iSCSI target using the command line interface <https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/>" I cannot start service rbd-target-api. There's no error message in status or anywhere else: root@ld5505:~# systemctl status rbd-target-api _ rbd-target-api.service - Ceph iscsi target configuration API Loaded: loaded (/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2019-12-04 13:47:51 CET; 3min 16s ago Process: 4143457 ExecStart=/usr/bin/rbd-target-api (code=exited, status=1/FAILURE) Main PID: 4143457 (code=exited, status=1/FAILURE) Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Service RestartSec=100ms expired, scheduling restart. Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Scheduled restart job, restart counter is at 3. Dec 04 13:47:51 ld5505 systemd[1]: Stopped Ceph iscsi target configuration API. Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Start request repeated too quickly. Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Failed with result 'exit-code'. Dec 04 13:47:51 ld5505 systemd[1]: Failed to start Ceph iscsi target configuration API. Can you please advise how to troubleshoot this issue? THX

2 4

What are the performance implications 'ceph fs set cephfs allow_new_snaps true'?
by Marc Roos 06 Dec '19

06 Dec '19

I recently enabled this and now my rsyncs are taking hours and hours longer.

1 0

Crushmap format in nautilus: documentation out of date
by Rodrigo Severo - Fábrica 06 Dec '19

06 Dec '19

Hi, The crushmap produced by ceph osd getcrushmap in ceph version 14.2.4 has more info than defined in https://docs.ceph.com/docs/cuttlefish/rados/operations/crush-map/ There is a second id per bucket: host a1-df { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 1.819 alg straw2 hash 0 # rjenkins1 item osd.0 weight 1.819 } What exactly is this second id which, by the way, comes with a device class specified? Regards, Rodrigo Severo

2 1

bluestore rocksdb behavior
by Frank R 06 Dec '19

06 Dec '19

Hi all, How is the following situation handled with bluestore: 1. You have a 200GB OSD (no separate DB/WAL devices) 2. The metadata grows past 30G for some reason and wants to create a 300GB level but can't? Where is the metadata over 30G stored?

3 4

iSCSI Gateway reboots and permanent loss
by Wesley Dillingham 05 Dec '19

05 Dec '19

We utilize 4 iSCSI gateways in a cluster and have noticed the following during patching cycles when we sequentially reboot single iSCSI-gateways: "gwcli" often hangs on the still-up iSCSI GWs but sometimes still functions and gives the message: "1 gateway is inaccessible - updates will be disabled" This got me thinking about what the course of action would be should an iSCSI gateway fail permanently or semi-permanently, say a hardware issue. What would be the best course of action to instruct the remaining iSCSI gateways that one of them is no longer available and that they should allow updates again and take ownership of the now-defunct-node's LUNS? I'm guessing pulling down the RADOS config object and rewriting it and re-put'ing it followed by a rbd-target-api restart might do the trick but am hoping there is a more "in-band" and less potentially devastating way to do this. Thanks for any insights. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

5 11

Re: 2 different ceph-users lists?
by Rodrigo Severo - Fábrica 05 Dec '19

05 Dec '19

Em qui., 5 de dez. de 2019 às 16:38, Marc Roos <M.Roos(a)f1-outsourcing.eu> escreveu: > > > > ceph-users(a)lists.ceph.com is old one, why this is, I also do not know Ok Marc. Thanks for your information. Rodrigo

1 0

2 different ceph-users lists?
by Rodrigo Severo - Fábrica 05 Dec '19

05 Dec '19

Hi, Are there 2 different ceph-users list? ceph-users(a)lists.ceph.com and ceph-users(a)ceph.io Why? What's the difference? Regards, Rodrigo Severo

2 1

Re: mds crash loop
by Karsten Nielsen 05 Dec '19

05 Dec '19

Thank you Yan, Zheng for the help to get my cephfs back in working order by providing a source version that had the fix in it to get the root inode fixed. (https://tracker.ceph.com/issues/42675) - Karsten -----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Tue 12-11-2019 11:55 Subject: Re: [ceph-users] Re: mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Tue, Nov 12, 2019 at 6:18 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > -----Original message----- > > From: Karsten Nielsen <karsten(a)foo-bar.dk> > > Sent: Tue 12-11-2019 10:30 > > Subject: [ceph-users] Re: mds crash loop > > To: Yan, Zheng <ukernel(a)gmail.com>; > > CC: ceph-users(a)ceph.io; > > > -----Original message----- > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > Sent: Mon 11-11-2019 15:09 > > > Subject: Re: [ceph-users] Re: mds crash loop > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > CC: ceph-users(a)ceph.io; > > > > On Mon, Nov 11, 2019 at 5:09 PM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > I started a job that moved some files around in the cephfs cluster that > > > > resulted in the mds to go back into the crash loop. > > > > > Logs are here: > > > > > http://s3.foo-bar.dk/mds-dumps/mds.log-20191111 > > > > > > > > > > Any help would be appriciated. > > > > > > > > > > > > > looks like snaptable is corrupted. > > > > > > > > nautilus version (14.2.2) of ‘cephfs-data-scan scan_links’ can fix > > > > snaptable. hopefully it will fix your issue. > > > > > > > > you don't need to upgrade whole cluster. Just install nautilus in a > > > > temp machine or compile ceph from source. > > > > > > I did run the command that you suggested, it did not unfortunately fix the > > > problem. > > > > > > http://s3.foo-bar.dk/mds-dumps/mds.log-20191112 > > > > > > > > > The output from the command is this: > > > > sudo docker exec -it rgw2 cephfs-data-scan scan_links > > 2019-11-12 08:46:27.025 7fe775dd7d80 -1 datascan.scan_links: Remove > duplicated ino 0x0x100026d17d4 from 0x100013b0d3d/latest.log > > 2019-11-12 08:46:28.665 7fe775dd7d80 -1 datascan.load_table: unable to read > mds table 'mds1_inotable': (2) No such file or directory > > 2019-11-12 08:46:28.665 7fe775dd7d80 -1 mds.1.inotable: erasing 0x20000000000 > to 0x2000000d665 > > 2019-11-12 08:46:28.793 7fe775dd7d80 -1 datascan.load_table: unable to read > mds table 'mds2_inotable': (2) No such file or directory > > 2019-11-12 08:46:28.793 7fe775dd7d80 -1 mds.2.inotable: erasing 0x30000000000 > to 0x300000228f5 > > 2019-11-12 08:46:29.345 7fe775dd7d80 -1 mds.0.snap updating last_snap 1 -> 3 > > > > > > please run ceph-mds with debug_mds=20, and send the crash log to me. > > Thanks > Yan, Zheng > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > -----Original message----- > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > Sent: Thu 07-11-2019 14:20 > > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > CC: ceph-users(a)ceph.io; > > > > > > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten(a)foo-bar.dk> > wrote: > > > > > > > > > > > > > > That is awesome. > > > > > > > > > > > > > > Now I just need to figure out where the lost+found files needs to > go. > > > > > > > And what happened to the missing objects for the dirs. > > > > > > > > > > > > > > > > > > > lost+found files are likely files that were deleted. you can keep the > > > > > > lost+found dir for a while, then delete the 'lost+found' directory. > > > > > > > > > > > > for 'missing object' dirs, mv all of them to a temp directory, such as > > > > > > /mnt/cephfs/missing_obj_dirs. > > > > > > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs > > > > > > force recursive repair'. wait a minute, the rm -rf > > > > > > /mnt/cephfs/missing_obj_dirs > > > > > > > > > > > > > Any tool that is able to do that ? > > > > > > > > > > > > > > Thanks > > > > > > > - Karsten > > > > > > > > > > > > > > -----Original message----- > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > Sent: Thu 07-11-2019 09:22 > > > > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > I have tracked down the root cause. See > > > > > > https://tracker.ceph.com/issues/42675 > > > > > > > > > > > > > > > > Regards > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen > <karsten(a)foo-bar.dk> > > > > wrote: > > > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > > Sent: Thu 07-11-2019 07:21 > > > > > > > > > Subject: Re: [ceph-users] Re: mds crash loop > > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen > > > <karsten(a)foo-bar.dk> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > > > > Sent: Wed 06-11-2019 14:16 > > > > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen > > > > <karsten(a)foo-bar.dk> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original message----- > > > > > > > > > > > > > From: Yan, Zheng <ukernel(a)gmail.com> > > > > > > > > > > > > > Sent: Wed 06-11-2019 08:15 > > > > > > > > > > > > > Subject: Re: [ceph-users] mds crash loop > > > > > > > > > > > > > To: Karsten Nielsen <karsten(a)foo-bar.dk>; > > > > > > > > > > > > > CC: ceph-users(a)ceph.io; > > > > > > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen > > > > > > <karsten(a)foo-bar.dk> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to > > > mimic > > > > > > 13.2.6 > > > > > > > > > > > > > > > It was running fine for a while but yesterday my mds > > > went > > > > > > into a > > > > > > > > crash > > > > > > > > > > > > loop. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs > both of > > > > which > > > > > > is > > > > > > > > > > running > > > > > > > > > > > > the > > > > > > > > > > > > > > same crash loop. > > > > > > > > > > > > > > > I am running ceph based on > > > > > > https://hub.docker.com/r/ceph/daemon > > > > > > > > > > version > > > > > > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv > > > > store. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Log details are: https://paste.debian.net/1113943/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > please try again with debug_mds=20. Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yan, Zheng > > > > > > > > > > > > > > > > > > > > > > > > > > Yes I have set that and had to move to pastebin.com as > > > debian > > > > > > > > apperently > > > > > > > > > > only > > > > > > > > > > > > supports 150k > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://pastebin.com/Gv7c5h54 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like on-disk root inode is corrupted. have you > > > > encountered any > > > > > > > > > > > > unusually things during the upgrade? > > > > > > > > > > > > > > > > > > > > > > > > please run 'rados -p <cephfs metadata pool> stat > > > > 1.00000000.inode' , > > > > > > > > > > > > check if the object is modified before or after the > 'luminous > > > -> > > > > > > > > > > > > 13.2.6' upgrade. > > > > > > > > > > > > To fix the corrupted object. Run 'cephfs-data-scan init > > > > > > > > > > > > --force-init'. Then restart mds. After mds become active, > run > > > > 'ceph > > > > > > > > > > > > daemon mds.x scrub_path / force repair' > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I followed the steps I got the mds started but now a lot of > > > files > > > > are > > > > > > in > > > > > > > > > > lost+found 24283 and I have these errors in the mds log > > > > > > > > > > > > > > > > > > > > > 'cephfs-data-scan init --force-init' does not move files into > > > > > > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' > > > command > > > > or > > > > > > > > > > 'cephfs-journal-tool foo' command? > > > > > > > > > > > > > > > > > > I have had a similar problem with the cluster before where I > went > > > > through > > > > > > the > > > > > > > > cycle of: > > > > > > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> > Using > > > an > > > > > > > > alternate metadata pool for recovery > > > > > > > > > > > > > > > > > > I did run the cephfs-journal-tool journal reset command, mostly > > > > because > > > > > > > > cephfs is not that utilized so I thought it was safe to do as > after > > > the > > > > > > upgrade > > > > > > > > the cluster has not been used much, so data lose would be minimal > - > > > > > > apparently > > > > > > > > I was wrong. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700 1 mds.0.32011 cluster > > > > recovered. > > > > > > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 0 > > > > mds.0.cache.dir(0x100013acfcb) > > > > > > > > > > _fetched missing object for [dir 0x100013acfcb > > > > > > > > /nextcloud/custom_apps/carnet/ > > > > > > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching > f() > > > n() > > > > > > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100] > > > > > > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 > log_channel(cluster) log > > > > > > [ERR] : > > > > > > > > dir > > > > > > > > > > 0x100013acfcb object missing on disk; some files may be lost > > > > > > > > > > (/nextcloud/custom_apps/carnet) > > > > > > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 0 > > > > mds.0.cache.dir(0x100013a3156) > > > > > > > > > > _fetched missing object for [dir 0x100013a3156 > > > > > > /nextcloud/custom_apps/mail/ > > > > > > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching > f() > > > n() > > > > > > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000] > > > > > > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 > log_channel(cluster) log > > > > > > [ERR] : > > > > > > > > dir > > > > > > > > > > 0x100013a3156 object missing on disk; some files may be lost > > > > > > > > > > (/nextcloud/custom_apps/mail) > > > > > > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 0 > > > > mds.0.cache.dir(0x100013abb3c) > > > > > > > > > > _fetched missing object for [dir 0x100013abb3c > > > > > > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 > > > ap=1+0+0 > > > > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 > > > authpin=1 > > > > > > > > > > 0x55d4dcc40700] > > > > > > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 > log_channel(cluster) log > > > > > > [ERR] : > > > > > > > > dir > > > > > > > > > > 0x100013abb3c object missing on disk; some files may be lost > > > > > > > > > > (/nextcloud/custom_apps/passwords) > > > > > > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 0 > > > > mds.0.cache.dir(0x100013a9b9b) > > > > > > > > > > _fetched missing object for [dir 0x100013a9b9b > > > > > > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 > > > ap=1+0+0 > > > > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 > > > authpin=1 > > > > > > > > > > 0x55d4dcc40e00] > > > > > > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 > log_channel(cluster) log > > > > > > [ERR] : > > > > > > > > dir > > > > > > > > > > 0x100013a9b9b object missing on disk; some files may be lost > > > > > > > > > > (/nextcloud/custom_apps/phonetrack) > > > > > > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 0 > > > > mds.0.cache.dir(0x100013a2659) > > > > > > > > > > _fetched missing object for [dir 0x100013a2659 > > > > > > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 > > > > ap=1+0+0 > > > > > > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 > > > authpin=1 > > > > > > > > > > 0x55d4dcc41500] > > > > > > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 > log_channel(cluster) log > > > > > > [ERR] : > > > > > > > > dir > > > > > > > > > > 0x100013a2659 object missing on disk; some files may be lost > > > > > > > > > > (/nextcloud/custom_apps/richdocuments) > > > > > > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700 1 mds.k8s-node-01 > > > Updating > > > > MDS > > > > > > map > > > > > > > > to > > > > > > > > > > version 32015 from mon.1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for any hints > > > > > > > > > > > > > > > - Karsten > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > > > > > > To unsubscribe send an email to > ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > >

1 0

Recommended procedure to modify Crush Map
by Thomas Schneider 05 Dec '19

05 Dec '19

Hi, my current Crush Map includes multiple roots representing different disks. There are multiple crush rules, one for each pool. And a pool represents a disk type: hdd, ssd, nvme Question: What is the recommended procedure to modify the Crush Map in order to define only one root and "transfer" all other roots to additional disk types? THX

1 0

Error in add new ISCSI gateway
by Gesiel Galvão Bernardes 04 Dec '19

04 Dec '19

Hi everyone, I have a problem trying to add an ISCSI gateway. The following error is generated when trying to add the new gateway: iscsi-target...-igw/gateways> create ceph-iscsi3 192.168.201.3 Adding gateway, sync'ing 3 disk(s) and 2 client(s) Failed : /etc/ceph/iscsi-gateway.cfg on ceph-iscsi3 does not match the local version. Correct and retry request But the file iscsi-gateway.cfg is exactly the same in all gateways. Selinux is disabled, permissios are OK too. I using ceph 13.2.6. Can anyone help-me? Regards Gesiel

3 10

Building a petabyte cluster from scratch
by Fabien Sirjean 04 Dec '19

04 Dec '19

Hi Ceph users ! After years of using Ceph, we plan to build soon a new cluster bigger than what we've done in the past. As the project is still in reflection, I'd like to have your thoughts on our planned design : any feedback is welcome :) ## Requirements * ~1 PB usable space for file storage, extensible in the future * The files are mostly "hot" data, no cold storage * Purpose : storage for big files being essentially used on windows workstations (10G access) * Performance is better :) ## Global design * 8+3 Erasure Coded pool * ZFS on RBD, exposed via samba shares (cluster with failover) ## Hardware * 1 rack (multi-site would be better, of course...) * OSD nodes : 14 x supermicro servers * 24 usable bays in 2U rackspace * 16 x 10 TB nearline SAS HDD (8 bays for future needs) * 2 x Xeon Silver 4212 (12C/24T) * 128 GB RAM * 4 x 40G QSFP+ * Networking : 2 x Cisco N3K 3132Q or 3164Q * 2 x 40G per server for ceph network (LACP/VPC for HA) * 2 x 40G per server for public network (LACP/VPC for HA) * QSFP+ DAC cables ## Sizing If we've done the maths well, we expect to have : * 2.24 PB of raw storage, extensible to 3.36 PB by adding HDD * 1.63 PB expected usable space with 8+3 EC, extensible to 2.44 PB * ~1 PB of usable space if we want to keep the OSD use under 66% to allow loosing nodes without problem, extensible to 1.6 PB (same condition) ## Reflections * We're used to run mons and mgrs daemons on a few of our OSD nodes, without any issue so far : is this a bad idea for a big cluster ? * We thought using cache tiering on an SSD pool, but a large part of the PB is used on a daily basis, so we expect the cache to be not so effective and really expensive ? * Could a 2x10G network be enough ? * ZFS on Ceph ? Any thoughts ? * What about CephFS ? We'd like to use RBD diff for backups but it looks impossible to use snapshot diff with Cephfs ? Thanks for reading, and sharing your experiences ! F.

12 16

Re: Can min_read_recency_for_promote be -1
by Romit Misra 03 Dec '19

03 Dec '19

Hi Robert, I am not quite sure if I get your question correct, but what I understand is that you want the inbound writes to land on the cache tier, which presumably would be on a faster media, possibily a ssd. From there you would want it to trickle down to the base tier, which is a EC pool hosted on HDD. Some of the pointers I have :- It is better to have seperate media for base and cache , HDD and SSD respectively. If the intent is never to promote to cache tier on Read, you could set it to a high number such as 3, and at the same time, Make the bloom filter window small.( This basically translates into if the object has been read X number of times in past y seconds) Keep in mind the larger the window, the more the size of the bloom filter, and hence you would see a increase in osd memory usgae. I have patch somewhere lurking which disables the promotes, let me check on the same, if this is for a specific case. If your intent is to have a constant decay rate from the Cache tier to the base tier, here is what you could do.:- 1.Set the Max Objects on the Cache tier TO X 2.Set the Max Size to say Y, this would be normally 60-70 percent of the total cache tier capacity. 3.The flushes would start happening on the first trigger of the above thresholds. 4. You could set the evict age roughly double the time, you expect the data will hit the base tier. 5.Lastly have you tried running cosbench or any related tool, to qualify the IOPS of your base tier with EC enabled, you may. Or require the cache tier at all. 6. There are substantial overheads of a cache tier maintenance, the major being absence of throttles on how the flush happens. 7.A thundering herd of write requests can cause a huge amount of flush to happen to the base tier. 8.IMHO it is suitable and predictable for loads where number of ingress requests can be predicted and there is some kind of rate limiting on the same. Hope this helps Thanks Romit On Tue, 3 Dec 2019, 04:11 , <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: ceph node crashed with these errors "kernel: ceph: > build_snap_context" (maybe now it is urgent?) > (Ilya Dryomov) > 2. Re: ceph node crashed with these errors "kernel: ceph: > build_snap_context" (maybe now it is urgent?) > (Marc Roos) > 3. Re: ceph node crashed with these errors "kernel: ceph: > build_snap_context" (maybe now it is urgent?) > (Marc Roos) > 4. Re: Possible data corruption with 14.2.3 and 14.2.4 > (Simon Ironside) > 5. Re: ceph node crashed with these errors "kernel: ceph: > build_snap_context" (maybe now it is urgent?) > (Marc Roos) > 6. Can min_read_recency_for_promote be -1 (Robert LeBlanc) > > > ---------------------------------------------------------------------- > > Date: Mon, 2 Dec 2019 14:59:05 +0100 > From: Ilya Dryomov <idryomov(a)gmail.com> > Subject: [ceph-users] Re: ceph node crashed with these errors "kernel: > ceph: build_snap_context" (maybe now it is urgent?) > To: Marc Roos <M.Roos(a)f1-outsourcing.eu> > Cc: ceph-users <ceph-users(a)ceph.io>, jlayton <jlayton(a)kernel.org> > Message-ID: > < > CAOi1vP-uyxeaKvuxUQbe2nsuXH9-f6_QxcggOCv6LrCBzugJOw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > On Mon, Dec 2, 2019 at 1:23 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > > > > I guess this is related? kworker 100% > > > > > > [Mon Dec 2 13:05:27 2019] SysRq : Show backtrace of all active CPUs > > [Mon Dec 2 13:05:27 2019] sending NMI to all CPUs: > > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 0 skipped: idling at pc > > 0xffffffffb0581e94 > > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 1 skipped: idling at pc > > 0xffffffffb0581e94 > > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 2 skipped: idling at pc > > 0xffffffffb0581e94 > > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 3 skipped: idling at pc > > 0xffffffffb0581e94 > > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 4 > > [Mon Dec 2 13:05:27 2019] CPU: 4 PID: 426200 Comm: kworker/4:2 Not > > tainted 3.10.0-1062.4.3.el7.x86_64 #1 > > [Mon Dec 2 13:05:27 2019] Hardware name: Supermicro > > X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.0b 05/27/2014 > > [Mon Dec 2 13:05:27 2019] Workqueue: ceph-msgr ceph_con_workfn > > [libceph] > > [Mon Dec 2 13:05:27 2019] task: ffffa0c8e1240000 ti: ffffa0ccb6364000 > > task.ti: ffffa0ccb6364000 > > [Mon Dec 2 13:05:27 2019] RIP: 0010:[<ffffffffc08d7db9>] > > [<ffffffffc08d7db9>] cmpu64_rev+0x19/0x20 [ceph] > > [Mon Dec 2 13:05:27 2019] RSP: 0018:ffffa0ccb6367a20 EFLAGS: 00000202 > > [Mon Dec 2 13:05:27 2019] RAX: 0000000000000001 RBX: 0000000000000038 > > RCX: 0000000000000008 > > [Mon Dec 2 13:05:27 2019] RDX: 0000000000025c33 RSI: ffffa0cbbe380050 > > RDI: ffffa0cbbe380030 > > [Mon Dec 2 13:05:27 2019] RBP: ffffa0ccb6367a20 R08: 0000000000000018 > > R09: 00000000000013ed > > [Mon Dec 2 13:05:27 2019] R10: 0000000000000002 R11: ffffe94994f8e000 > > R12: ffffa0cbbe380030 > > [Mon Dec 2 13:05:27 2019] R13: ffffffffc08d7da0 R14: ffffa0cbbe380018 > > R15: ffffa0cbbe380050 > > [Mon Dec 2 13:05:27 2019] FS: 0000000000000000(0000) > > GS:ffffa0d2cfb00000(0000) knlGS:0000000000000000 > > [Mon Dec 2 13:05:27 2019] CS: 0010 DS: 0000 ES: 0000 CR0: > > 0000000080050033 > > [Mon Dec 2 13:05:27 2019] CR2: 000055a7c413fcb9 CR3: 0000001813010000 > > CR4: 00000000000607e0 > > [Mon Dec 2 13:05:27 2019] Call Trace: > > [Mon Dec 2 13:05:27 2019] [<ffffffffb019303f>] sort+0x1af/0x260 > > [Mon Dec 2 13:05:27 2019] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 > > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d807b>] > > build_snap_context+0x12b/0x290 [ceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d820c>] > > rebuild_snap_realms+0x2c/0x90 [ceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d822b>] > > rebuild_snap_realms+0x4b/0x90 [ceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffc08d91fc>] > > ceph_update_snap_trace+0x3ec/0x530 [ceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffc08e2239>] > > handle_reply+0x359/0xc60 [ceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 > > [ceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffb042e56a>] ? > > kernel_recvmsg+0x3a/0x50 > > [Mon Dec 2 13:05:27 2019] [<ffffffffc05fcff4>] try_read+0x544/0x1300 > > [libceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffafee13ce>] ? > > account_entity_dequeue+0xae/0xd0 > > [Mon Dec 2 13:05:27 2019] [<ffffffffafee4d5c>] ? > > dequeue_entity+0x11c/0x5e0 > > [Mon Dec 2 13:05:27 2019] [<ffffffffb042e417>] ? > > kernel_sendmsg+0x37/0x50 > > [Mon Dec 2 13:05:27 2019] [<ffffffffc05fdfb4>] > > ceph_con_workfn+0xe4/0x1530 [libceph] > > [Mon Dec 2 13:05:27 2019] [<ffffffffb057f568>] ? > > __schedule+0x448/0x9c0 > > [Mon Dec 2 13:05:27 2019] [<ffffffffafebe21f>] > > process_one_work+0x17f/0x440 > > [Mon Dec 2 13:05:27 2019] [<ffffffffafebf336>] > > worker_thread+0x126/0x3c0 > > [Mon Dec 2 13:05:27 2019] [<ffffffffafebf210>] ? > > manage_workers.isra.26+0x2a0/0x2a0 > > [Mon Dec 2 13:05:27 2019] [<ffffffffafec61f1>] kthread+0xd1/0xe0 > > [Mon Dec 2 13:05:27 2019] [<ffffffffafec6120>] ? > > insert_kthread_work+0x40/0x40 > > [Mon Dec 2 13:05:27 2019] [<ffffffffb058cd37>] > > ret_from_fork_nospec_begin+0x21/0x21 > > [Mon Dec 2 13:05:27 2019] [<ffffffffafec6120>] ? > > insert_kthread_work+0x40/0x40 > > [Mon Dec 2 13:05:27 2019] Code: 87 c8 fc ff ff 5d 0f 94 c0 0f b6 c0 c3 > > 0f 1f 44 00 00 66 66 66 66 90 48 8b 16 48 39 17 b8 01 00 00 00 55 48 89 > > e5 72 08 0f 97 c0 <0f> b6 c0 f7 d8 5d c3 66 66 66 66 90 55 f6 05 ed 92 > > 02 00 04 48 > > [Mon Dec 2 13:05:27 2019] NMI backtrace for cpu 5 > > Yes, seems related. I'm not sure how it relates to an upgrade to > nautilus, but as I mentioned in a different message, with thousands of > snapshots you are in a dangerous territory anyway. > > Thanks, > > Ilya > > ------------------------------ > > Date: Mon, 2 Dec 2019 15:06:54 +0100 > From: "Marc Roos" <M.Roos(a)f1-outsourcing.eu> > Subject: [ceph-users] Re: ceph node crashed with these errors "kernel: > ceph: build_snap_context" (maybe now it is urgent?) > To: idryomov <idryomov(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io>, jlayton <jlayton(a)kernel.org> > Message-ID: <"H000007100158998.1575295614.sx.f1-outsourcing.eu*"@MHS> > Content-Type: text/plain; charset="UTF-8" > > > > >> > > >> >ISTR there were some anti-spam measures put in place. Is your > account > >> >waiting for manual approval? If so, David should be able to help. > >> > >> Yes if I remember correctly I get waiting approval when I try to log > in. > >> > >> >> > >> >> > >> >> > >> >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287 > >> >> ffff911a9a26bd00 fail -12 > >> >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9283 > >> > > >> > > >> >It is failing to allocate memory. "low load" isn't very specific, > >> >can you describe the setup and the workload in more detail? > >> > >> 4 nodes (osd, mon combined), the 4th node has local cephfs mount, > which > >> is rsync'ing some files from vm's. 'low load' I have sort of test > setup, > >> going to production. Mostly the nodes are below a load of 1 (except > when > >> the concurrent rsync starts) > >> > >> >How many snapshots do you have? > >> > >> Don't know how to count them. I have script running on a 2000 dirs. > If > >> one of these dirs is not empty it creates a snapshot. So in theory I > >> could have 2000 x 7 days = 14000 snapshots. > >> (btw the cephfs snapshots are in a different tree than the rsync is > >> using) > > > >Is there a reason you are snapshotting each directory individually > >instead of just snapshotting a common parent? > > Yes because I am not sure the snapshot frequency on all folders is going > to be the same. > > >If you have thousands of snapshots, you may eventually hit a different > >bug: > > > >https://tracker.ceph.com/issues/21420 > >https://docs.ceph.com/docs/master/cephfs/experimental-features/#snapsh > ots > > > >Be aware that each set of 512 snapshots amplify your writes by 4K in > >terms of network consumption. With 14000 snapshots, a 4K write would > >need to transfer ~109K worth of snapshot metadata to carry itself out. > > > > Also when I am not even writing to a tree with snapshots enabled? I am > rsyncing to dir3 > > . > ├── dir1 > │ ├── dira > │ │ └── .snap > │ ├── dirb > │ ├── dirc > │ │ └── .snap > │ └── dird > │ └── .snap > ├── dir2 > └── dir3 > > ------------------------------ > > Date: Mon, 2 Dec 2019 16:29:07 +0100 > From: "Marc Roos" <M.Roos(a)f1-outsourcing.eu> > Subject: [ceph-users] Re: ceph node crashed with these errors "kernel: > ceph: build_snap_context" (maybe now it is urgent?) > To: idryomov <idryomov(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.io>, jlayton <jlayton(a)kernel.org> > Message-ID: <"H000007100158aca.1575300547.sx.f1-outsourcing.eu*"@MHS> > Content-Type: text/plain; charset="UTF-8" > > > I can confirm that removing all the snapshots seems to resolve the > problem. > > A - I would propose a redesign of something like that snapshots from > below the mountpoint are only taken into account and not snapshots in > the entire filesystem. That should fix a lot of issues > > B - That reminds me about this mv command, that does not move data > across different pools in the fs. I would like to see this. Because it > is the logical thing to expect. > > > > > > > >> > > >> >ISTR there were some anti-spam measures put in place. Is your > account >> >waiting for manual approval? If so, David should be able > to help. > >> > >> Yes if I remember correctly I get waiting approval when I try to log > in. > >> > >> >> > >> >> > >> >> > >> >> Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287 > >> >> ffff911a9a26bd00 fail -12 >> >> Dec 1 03:14:36 c04 kernel: > ceph: build_snap_context 100020c9283 >> > >> > >> >It is failing > to allocate memory. "low load" isn't very specific, >> >can you > describe the setup and the workload in more detail? > >> > >> 4 nodes (osd, mon combined), the 4th node has local cephfs mount, > which >> is rsync'ing some files from vm's. 'low load' I have sort of > test setup, >> going to production. Mostly the nodes are below a load > of 1 (except when >> the concurrent rsync starts) >> >> >How many > snapshots do you have? > >> > >> Don't know how to count them. I have script running on a 2000 dirs. > If > >> one of these dirs is not empty it creates a snapshot. So in theory I > >> could have 2000 x 7 days = 14000 snapshots. > >> (btw the cephfs snapshots are in a different tree than the rsync is > >> using) > >Is there a reason you are snapshotting each directory > individually >instead of just snapshotting a common parent? > > Yes because I am not sure the snapshot frequency on all folders is going > to be the same. > > >If you have thousands of snapshots, you may eventually hit a different > >bug: > > > >https://tracker.ceph.com/issues/21420 > >https://docs.ceph.com/docs/master/cephfs/experimental-features/#snapsh > ots > > > >Be aware that each set of 512 snapshots amplify your writes by 4K in > >terms of network consumption. With 14000 snapshots, a 4K write would > >need to transfer ~109K worth of snapshot metadata to carry itself out. > > > > Also when I am not even writing to a tree with snapshots enabled? I am > rsyncing to dir3 > > . > ├── dir1 > │ ├── dira > │ │ └── .snap > │ ├── dirb > │ ├── dirc > │ │ └── .snap > │ └── dird > │ └── .snap > ├── dir2 > └── dir3 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > > ------------------------------ > > Date: Mon, 2 Dec 2019 15:54:54 +0000 > From: Simon Ironside <sironside(a)caffetine.org> > Subject: [ceph-users] Re: Possible data corruption with 14.2.3 and > 14.2.4 > To: ceph-users(a)ceph.io > Message-ID: <21d057e9-0088-4847-6d40-19cf2c848395(a)caffetine.org> > Content-Type: text/plain; charset=utf-8; format=flowed > > Any word on 14.2.5? Nervously waiting here . . . > > Thanks, > Simon. > > On 18/11/2019 11:29, Simon Ironside wrote: > > > I will sit tight and wait for 14.2.5. > > > > Thanks again, > > Simon. > > ------------------------------ > > Date: Mon, 2 Dec 2019 19:32:03 +0100 > From: "Marc Roos" <M.Roos(a)f1-outsourcing.eu> > Subject: [ceph-users] Re: ceph node crashed with these errors "kernel: > ceph: build_snap_context" (maybe now it is urgent?) > To: ceph-users <ceph-users(a)ceph.io>, lhenriques <lhenriques(a)suse.com> > Message-ID: <"H000007100158b41.1575311519.sx.f1-outsourcing.eu*"@MHS> > Content-Type: text/plain; charset="ISO-8859-1" > > > Yes Luis, good guess!! ;) > > > > -----Original Message----- > Cc: ceph-users > Subject: Re: [ceph-users] ceph node crashed with these errors "kernel: > ceph: build_snap_context" (maybe now it is urgent?) > > On Mon, Dec 02, 2019 at 10:27:21AM +0100, Marc Roos wrote: > > > > I have been asking before[1]. Since Nautilus upgrade I am having > > these, with a total node failure as a result(?). Was not expecting > > this in my 'low load' setup. Maybe now someone can help resolving > > this? I am also waiting quite some time to get access at > > https://tracker.ceph.com/issues. > > Just a wild guess: do you have a lot of snapshots (> ~400)? If so, > that's probably the problem. See [1] and [2]. > > [1] > https://docs.ceph.com/docs/master/cephfs/experimental-features/#snapshots > [2] https://tracker.ceph.com/issues/21420 > > Cheers, > -- > Luís > > > > > > > Dec 1 03:14:36 c04 kernel: ceph: build_snap_context 100020c9287 > > ffff911a9a26bd00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9283 ffff911d34e69d00 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9276 > > ffff911d34e69c00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c926c ffff912068b92c00 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9268 > > ffff912068b93000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c926d ffff912068b92900 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c928a > > ffff912118e5be00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9272 ffff9119950d9500 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9269 > > ffff911940f3d000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9270 ffff911748427c00 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c926b > > ffff91169b000600 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9281 ffff91169b000500 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9288 > > ffff9115844d2500 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c927d ffff9115844d2e00 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9280 > > ffff91186401b000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9267 ffff9121535ecc00 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c927c > > ffff9121cecb1e00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9271 ffff9121cecb0400 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9279 > > ffff911d26646300 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c927f ffff911d26646900 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9275 > > ffff9121cecb1700 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9259 ffff91170c9f6600 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9257 > > ffff9118ef2a8000 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c924e ffff911a1e091800 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9262 > > ffff911a1e090c00 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9266 ffff9115e3859500 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c924f > > ffff9118aefd1300 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c925f ffff91170c9f6100 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9252 > > ffff9115e3859800 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9256 ffff912045dc5300 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020c9254 > > ffff91170c9f6900 fail -12 Dec 1 03:14:36 c04 kernel: ceph: > > build_snap_context 100020c9261 ffff91170c9f7100 fail -12 Dec 1 > > 03:14:36 c04 kernel: ceph: build_snap_context 100020d4ec4 > > ffff9118aefd0000 fail -12 > > > > [1] > > https://www.mail-archive.com/ceph-users@ceph.io/msg01088.html > > https://www.mail-archive.com/ceph-users@ceph.io/msg00969.html > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > > ------------------------------ > > Date: Mon, 2 Dec 2019 14:39:26 -0800 > From: Robert LeBlanc <robert(a)leblancnet.us> > Subject: [ceph-users] Can min_read_recency_for_promote be -1 > To: ceph-users <ceph-users(a)ceph.io> > Message-ID: > < > CAANLjFoecdW7oBh78L3dNO83C-DpDmqXw-kKtT+ShNKXjsqKJg(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="00000000000024d4be0598c041cd" > > --00000000000024d4be0598c041cd > Content-Type: text/plain; charset="UTF-8" > > I'd like to configure a cache tier to act as a write buffer, so that if > writes come in, it promotes objects, but reads never promote an object. We > have a lot of cold data so we would like to tier down to an EC pool > (CephFS) after a period of about 30 days to save space. The storage tier > and the 'cache' tier would be on the same spindles, so the only performance > improvement would be from the faster writes with replication. So we don't > want to really move data between tiers. > > The idea would be to not promote on read since EC read performance is good > enough and have writes go to the cache tier where the data may be 'hot' for > a week or so, then get cold. > > It seems that we would only need one hit_set and if -1 can't be set for > min_read_recency_for_promote, I could probably use 2 which would never hit > because there is only one set, but that may error too. The follow up is how > big a set should be as it only really tells if an object "may" be in cache > and does not determine when things are flushed, so it really only matters > how out-of-date we are okay with the bloom filter being out of date, right? > So we could have it be a day long if we are okay with that stale rate? Is > there any advantage to having a longer period for a bloom filter? Now, I'm > starting to wonder if I even need a bloom filter for this use case, can I > get tiering to work without it and only use > cache_min_flush_age/cach_min_evict_age since I don't care about promoting > when there are X hits in Y time? > > Thanks > ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > --00000000000024d4be0598c041cd > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr">I'd like to configure a cache tier to act as a write > b= > uffer, so that if writes come in, it promotes objects, but reads never > prom= > ote an object. We have a lot of cold data so we would like to tier down to > = > an EC pool (CephFS) after a period of about 30 days to save space. The > stor= > age tier and the 'cache' tier would be on the same spindles, so > the= > only performance improvement would be from the faster writes with > replicat= > ion. So we don't want to really move data between > tiers.<div><br></div>= > <div>The idea would be to not promote on read since EC read performance is > = > good enough and have writes go to the cache tier where the data may be > &#39= > ;hot' for a week or so, then get cold.</div><div><br></div><div>It > seem= > s that we would only need one hit_set and if -1 can't be set for > min_re= > ad_recency_for_promote, I could probably use 2 which would never hit > becaus= > e there is only one set, but that may error too. The follow up is how big > a= > set should be as it only really tells if an object "may" be in > c= > ache and does not determine when things are flushed, so it really only > matt= > ers how out-of-date we are okay with the bloom filter being out of date, > ri= > ght? So we could have it be a day long if we are okay with that stale > rate?= > Is there any advantage to having a longer period for a bloom filter? Now, > = > I'm starting to wonder if I even need a bloom filter for this use > case,= > can I get tiering to work without it and only use > cache_min_flush_age/cach= > _min_evict_age since I don't care about promoting when there are X > hits= > in Y time?</div><div><br></div><div>Thanks<br clear=3D"all"><div><div dir= > =3D"ltr" class=3D"gmail_signature" > data-smartmail=3D"gmail_signature">-----= > -----------<br>Robert LeBlanc<br>PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 = > =C2=A0C70E E654 3BB2 FA62 B9F1</div></div></div></div> > > --00000000000024d4be0598c041cd-- > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 83, Issue 5 > ***************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

2 1

Behavior of EC pool when a host goes offline
by majia xiao 03 Dec '19

03 Dec '19

Hi all, We have a Ceph（version 12.2.4）cluster that adopts EC pools, and it consists of 10 hosts for OSDs. The corresponding commands to create the EC pool are listed as follows: ceph osd erasure-code-profile set profile_jerasure_4_3_reed_sol_van \ plugin=jerasure \ k=4 \ m=3 \ technique=reed_sol_van \ packetsize=2048 \ crush-device-class=hdd \ crush-failure-domain=host ceph osd pool create pool_jerasure_4_3_reed_sol_van 2048 2048 erasure profile_jerasure_4_3_reed_sol_van Since that the EC pool's crush-failure-domain is configured to be "host", we just disable the network interfaces of some hosts (using "ifdown" command) to verify the functionality of the EC pool. And here are the phenomena we have observed: First of all, the IO rate (of "rados bench", which we used for benchmark) drops immediately to 0 when one host goes offline. Secondly, it takes a lot of time (around 100 seconds) for Ceph to detect corresponding OSDs on that host are down. Finally, once the Ceph has detected all offline OSDs, the EC pool seems to act normally and it is ready for IO operations again. So, here are my questions: 1. Is this normal that the IO rate drops to 0 immediately even though there is only one host goes offline? 2. How to make Ceph reduce the time needed to detect failed OSDs? Thanks for any help. Best regards, Majia Xiao

2 1

how to speed up mount a ceph fs when a node unusual down in ceph cluster
by hfx＠portsip.cn 03 Dec '19

03 Dec '19

I created one ceph cluster. node-1: mon, mgr, osd.0, mds node-2: mon, mgr, osd.1, mds node-3: mon, mgr, osd.2, mds When the cluster is working normally, using command "mount -t ceph <node-*-ip:6789>:/ /mnt -o name=admin,secret=<admin client secret>" to mount is ok. But when a node unusual down(like poweroff), and using same command to mount, it will be hanging long time(maybe more then 1 minutes). I trying to configure "mds reconnect timeout = 0" in ceph.conf, mount time has been shortened. My Question: What are the configurations that affect the ceph file system mount in this scenario? BRs. hfx(a)portsip.cn

1 0

Possible data corruption with 14.2.3 and 14.2.4
by Sage Weil 03 Dec '19

03 Dec '19

Hi everyone, We've identified a data corruption bug[1], first introduced[2] (by yours truly) in 14.2.3 and affecting both 14.2.3 and 14.2.4. The corruption appears as a rocksdb checksum error or assertion that looks like os/bluestore/fastbmap_allocator_impl.h: 750: FAILED ceph_assert(available >= allocated) or in some cases a rocksdb checksum error. It only affects BlueStore OSDs that have a separate 'db' or 'wal' device. We have a fix[3] that is working its way through testing, and will expedite the next Nautilus point release (14.2.5) once it is ready. If you are running 14.2.2 or 14.2.1 and use BlueStore OSDs with separate 'db' volumes, you should consider waiting to upgrade until 14.2.5 is released. A big thank you to Igor Fedotov and several *extremely* helpful users who managed to reproduce and track down this problem! sage [1] https://tracker.ceph.com/issues/42223 [2] https://github.com/ceph/ceph/commit/096033b9d931312c0688c2eea7e14626bfde0ad… [3] https://github.com/ceph/ceph/pull/31621

5 8

Can min_read_recency_for_promote be -1
by Robert LeBlanc 03 Dec '19

03 Dec '19

I'd like to configure a cache tier to act as a write buffer, so that if writes come in, it promotes objects, but reads never promote an object. We have a lot of cold data so we would like to tier down to an EC pool (CephFS) after a period of about 30 days to save space. The storage tier and the 'cache' tier would be on the same spindles, so the only performance improvement would be from the faster writes with replication. So we don't want to really move data between tiers. The idea would be to not promote on read since EC read performance is good enough and have writes go to the cache tier where the data may be 'hot' for a week or so, then get cold. It seems that we would only need one hit_set and if -1 can't be set for min_read_recency_for_promote, I could probably use 2 which would never hit because there is only one set, but that may error too. The follow up is how big a set should be as it only really tells if an object "may" be in cache and does not determine when things are flushed, so it really only matters how out-of-date we are okay with the bloom filter being out of date, right? So we could have it be a day long if we are okay with that stale rate? Is there any advantage to having a longer period for a bloom filter? Now, I'm starting to wonder if I even need a bloom filter for this use case, can I get tiering to work without it and only use cache_min_flush_age/cach_min_evict_age since I don't care about promoting when there are X hits in Y time? Thanks ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 1